Use Context
Benchmarks and monitors
A benchmark runs a dataset on demand to compare models, skills or agents. A monitor grades a sample of real requests as they finish.
Comparison
| Benchmark | Monitor | |
|---|---|---|
| What it grades | A dataset of tasks you chose | Real requests, as they finish |
| When it runs | When you click Run | All the time, on a sample of requests |
| Use it to | Test a change before you ship it: a new model, skill or set of instructions | Notice when quality drops after something is live |
| Compares | Models, skills or agents, side by side | Nothing: it watches one or more targets |
| What it watches | The agent, model and skills you hold fixed | Targets: a skill, agent, applet, workflow, or all traces |
| Criteria | Any kind, including exact match against the expected answer | An LLM judge, an agent judge or a Python check. No exact match, because a real request has no expected answer |
Benchmarks and monitors can target the same agent: a benchmark before a change ships, a monitor after.
Create a benchmark
- Go to Evals > Evaluations, click New evaluation and choose Benchmark. To start from an agent, skill or workflow instead, open its Evaluations tab and use New evaluation there; it fills that in for you.
- Fill in the page from the top:
- a name;
- Compare: Models, Skills or Agents. Then pick the ones to compare, and what stays the same for every answer (the agent, model or skills held);
- Dataset: click Add dataset and pick one, or choose Upload a file… to import one;
- Pass mark: it starts at 75%.
- Check the tasks and criteria that come with the dataset. To change them, click Edit them in the dataset.
- Click Create and run. The benchmark is saved, its first run starts, and it opens on Activity.
The first run tries every task once for each model, skill or agent you compare. A run grades only the dataset’s saved version, so if it has unsaved changes you’re asked whether to save them first. A new benchmark grades with your organization’s default judge model; once it’s saved, you can change the Judge model on its Editor tab.
Create a monitor
- On Evals > Evaluations, click New evaluation and choose Monitor. You can also start from the Evaluations tab of an agent, skill, applet or workflow. If nothing checks it yet, that tab offers a Check name’s answers button.
- Fill in:
- Targets: what it watches. Click Add target to add an agent, skill or workflow. To watch an applet, start from the applet’s Evaluations tab. Once the monitor is saved, you can also add applets or All traces;
- Sampling: the percentage of requests it grades, from 1 to 100. The same request is always in or always out;
- Pass mark (it starts at 85%) and Judge model;
- at least one criterion. Each is an LLM judge, an agent judge or a Python check. A Python check needs its own pass mark. An agent judge doesn’t run on applet or workflow targets.
- Click Create and start grading. Grading starts with the next request, so a new monitor shows nothing until then.
How a monitor grades
- A monitor watches up to 50 targets and grades with up to 50 criteria, each with a weight.
- A graded request passes when the weighted score of its criteria reaches the pass mark.
- A monitor can’t use an exact match criterion, because a live request has no expected answer to match.
- A criterion that can’t reach a verdict, for example because its judge ran out of time, reads “Couldn’t be judged”. It is left out of the score and the pass rate.
- A monitor’s checks are charged to one person: whoever last created, changed or resumed it.
Monitoring has to be turned on for your organization. If the Evaluations page says “Monitoring is off for this organization”, ask an organization admin. Benchmarks still work.
Next steps
- Dataset files: import the tasks and criteria a benchmark runs.
- Criteria: write criteria that grade the same way every time.
- Read results: read a benchmark’s or monitor’s Activity tab.