Write criteria
A criterion is one requirement an answer must meet. A dataset’s criteria together form its rubric, and the judge applies each one literally.
Criterion types
| Kind | What it does | Use it for |
|---|---|---|
| LLM judge | One model reads the answer and the criterion, and gives a pass or fail or a score from 0 to 1 | Most criteria: tone, completeness, facts |
| Agent judge | An agent opens the answer, the expected answer and any files, then gives a pass or fail or a score from 0 to 1. Slower and costs more than an LLM judge | Answers that are files, such as a spreadsheet or deck, or comparing with an expected file |
| Exact match | No model. The answer equals the expected answer exactly, case and spaces included, or contains or leaves out a given text, or the agent called a named tool | Short fixed answers such as a date, an ID or a number; a phrase that must or must not appear; a tool that must be used |
| Python check | Your own code returns a pass or fail, or a score from 0 to 1 | Anything you can count: length, format, a value in a file |
Python checks need your administrator to turn them on. They run in a sandbox when the evaluation runs. The editor can’t try a Python check on a task, so run the benchmark to see how it grades.
A dataset task with expected files can be graded only by an agent judge or a Python check, because those are the two kinds that open files. That includes the dataset’s criteria that grade every dataset task.
A monitor can’t use an exact match, because a real request has no expected answer.
Criteria on a dataset, agent judges included, need the task engine your deployment uses by default. If your administrator switched to the older engine, saving criteria is refused.
Weights
Weights are relative and don’t need to add up to 100. A task’s score is the weighted average of its criteria. A criterion that couldn’t be judged on a task is left out of that task’s score instead of counting as a fail.
How a task is graded
Each dataset task is graded one of two ways:
- Expected answer: one right answer, as text, files or both, and the score is how well the answer matches it.
- Rubric: weighted criteria the answer must meet.
The dataset’s own criteria (the Shared rubric on a benchmark) grade every task as well, whichever way it is graded. A task with a rubric adds its own criteria on top.
A task can also be Held out. It’s graded like any other task, but the engine never learns from it, so you can check that an improvement wasn’t fitted to the tasks it came from. You set it on a benchmark’s tasks, not in a dataset file.
Best practices
- Make each criterion checkable. The judge applies the words literally, so a vague criterion gives an unreliable score. The table below shows vague criteria and checkable rewrites.
- Put one requirement in each criterion. “Apologizes and gives the new date” is two criteria; if the answer does one and not the other, a combined criterion can’t say which part failed.
- Weight what matters most. Give the criteria that would make an answer unusable, such as an invented fact, more weight than style.
| Instead of | Write |
|---|---|
| Is concise | Is 150 words or fewer |
| Is accurate | Does not state any number or fact that is not in the prompt or attached files |
| Mentions the key details | States the new delivery date (14 November) |
| Sounds professional | Uses plain customer-facing language with no internal codes or warehouse names |
Next steps
- Dataset files: import criteria with a dataset’s tasks, starting from the rubric template.
- Benchmarks and monitors: create an evaluation that grades with these criteria.
- Read results: read each criterion’s pass rate on a run’s scorecard.