Reading results
A saved benchmark or monitor has two tabs: Editor, where you change what it tests and how it’s graded, and Activity, where its results are.
A benchmark’s Activity
Four figures at the top:
| Figure | What it means |
|---|---|
| Best score | The highest score any finished run reached, and which model, skill or agent got it |
| Runs | How many times the benchmark has run |
| Above pass mark | How many finished runs scored at or above the pass mark as it is set now |
| Cost | What the last run that ended cost, and its cost per task |
Below them, one chart for the last 20 finished runs. Its buttons switch between Score by run, Score by candidate and Cost by run. The score line breaks where the compared models, skills or agents, or the dataset version, changed, because scores on either side aren’t comparable.
Then Runs: a run still in progress comes first, with its progress and Stop, then every run, newest first, with its score, cost and status. Change is how many points the score moved from the previous run that scored. Click a run to open it.
A run
The header shows the run’s score, the benchmark’s name, and the run’s state. Around it are Export, Stop while it runs, and Run again.
A run tries every task once for each model, skill or agent compared, and again for each repeat you set with Runs per task. Each of those tries is one run of the task.
Below is the run’s scorecard:
- Leaderboard: one row per model, skill or agent compared, best first. It shows Score (average) over every run of every task, Score (max), the best single run, and Cost / attempt and p50, the median time an answer took. With a pass mark it also shows pass@N: the share of tasks where at least one of the N repeats passed.
- Tasks, grouped by how each task did across the run: Fails everywhere and Passes everywhere (every run on every model, skill or agent), Unstable (repeats of the same one disagree), Split (each one is consistent, but they differ), and Not settled while grading is still going. These groups need a pass mark. Switch to Criteria to read each criterion’s pass rate on each model, skill or agent.
Click a task to open it. There’s one tab per run of that task, one for each model, skill or agent and repeat, each with the input, the answer and how every criterion scored it.
A monitor’s Activity
A monitor grades a sample of real requests as they finish, so its Activity shows how they’ve been doing. The four figures at the top cover the last 24 hours: Pass rate, Graded, Failing and Cost. The chart covers the last 7 days; switch it between Pass rate, Criterion and Cost.
Runs lists every graded request, newest first, with any still being graded at the top. Each is Passed, Failed or Not graded; switch to Failed to see only the failures. A criterion that couldn’t be judged is left out of the request’s score.
Run it again
Click Run on a benchmark that has run before, or Run again on a run. The dialog shows what’s Changed since the last run, or says “Nothing changed: this re-runs the same setup”. From there you can set Runs per task (1 to 5), run a sample of the tasks instead of all of them, and see the size and expected cost of the run before you start.
What to do with a score
- Start with the tasks that fail everywhere. They usually point at one missing instruction or a missing file.
- Treat Unstable tasks separately. A task that passes on some runs and fails on others needs a clearer prompt or criterion, not a different model. Set Runs per task above 1 to see which tasks are unstable.
- Change one thing between runs. If you change the instructions and the model together, you won’t know which one moved the score.