Benchmark or monitor
The Evaluations page puts it in one line: benchmarks test before you ship; monitors grade real requests in production.
| Benchmark | Monitor | |
|---|---|---|
| What it grades | A dataset of tasks you chose | Real requests, as they finish |
| When it runs | When you click Run | All the time, on a sample of requests |
| Use it to | Test a change before you ship it: a new model, skill or set of instructions | Notice when quality drops after something is live |
| Compares | Models, skills or agents, side by side | Nothing: it watches one or more targets |
| What it watches | The agent, model and skills you hold fixed | Targets: a skill, agent, applet, workflow, or all traces |
| Criteria | Any kind, including exact match against the expected answer | An LLM judge, an agent judge or a Python check. No exact match, because a real request has no expected answer |
Most teams use both: a benchmark to decide whether a change is ready, then a monitor on the same agent once it’s live.
Create a benchmark
- Go to Evals > Evaluations, click New evaluation and choose Benchmark. To start from an agent, skill or workflow instead, open its Evaluations tab and use New evaluation there; it fills that in for you.
- Fill in the page from the top:
- a name;
- Compare: Models, Skills or Agents. Then pick the ones to compare, and what stays the same for every answer (the agent, model or skills held);
- Dataset: click Add dataset and pick one, or choose Upload a file… to import one;
- Pass mark: it starts at 75%.
- Check the tasks and criteria that come with the dataset. To change them, click Edit them in the dataset.
- Click Create and run. The benchmark is saved, its first run starts, and it opens on Activity.
The first run runs every task once for each model, skill or agent you compare. A run grades only the dataset’s saved version, so if it has unsaved changes you’re asked whether to save them first. A new benchmark grades with your organization’s default judge model; once it’s saved, you can change the Judge model on its Editor tab.
Create a monitor
- On Evals > Evaluations, click New evaluation and choose Monitor. You can also start from the Evaluations tab of an agent, skill, applet or workflow. If nothing checks it yet, that tab offers a Check name’s answers button.
- Fill in:
- Targets: what it watches. Click Add target to add an agent, skill or workflow. To watch an applet, start from the applet’s Evaluations tab. Once the monitor is saved, you can also add applets or All traces;
- Sampling: the percentage of requests it grades, from 1 to 100. The same request is always in or always out;
- Pass mark (it starts at 85%) and Judge model;
- at least one criterion. Each is an LLM judge, an agent judge or a Python check. A Python check needs its own pass mark. An agent judge doesn’t run on applet or workflow targets.
- Click Create and start grading. Grading starts with the next request, so a new monitor shows nothing until then.
A criterion that can’t reach a verdict, for example because its judge ran out of time, reads “Couldn’t be judged”. It is left out of the score and the pass rate.
Monitoring has to be turned on for your organization. If the Evaluations page says “Monitoring is off for this organization”, ask an organization admin. Benchmarks still work.