Writing criteria that grade well
A criterion says one thing a good answer must do. The judge reads it word for word, so the way you write it decides how reliable the score is.
Make each one checkable
| Instead of | Write |
|---|---|
| Is concise | Is 150 words or fewer |
| Is accurate | Does not state any number or fact that is not in the prompt or attached files |
| Mentions the key details | States the new delivery date (14 November) |
| Sounds professional | Uses plain customer-facing language with no internal codes or warehouse names |
One idea per criterion
“Apologizes and gives the new date” is two criteria. If the answer does one and not the other, a combined criterion can’t say which part failed.
Pick the right kind of check
| Kind | What it does | Use it for |
|---|---|---|
| LLM judge | One model reads the answer and the criterion, and gives a pass or fail or a score from 0 to 1 | Most criteria: tone, completeness, facts |
| Agent judge | An agent opens the answer, the expected answer and any files, then gives a pass or fail or a score from 0 to 1. Slower and costs more than an LLM judge | Answers that are files, such as a spreadsheet or deck, or comparing with an expected file |
| Exact match | No model. The answer equals the expected answer exactly, case and spaces included, or contains or leaves out a given text, or the agent called a named tool | Short fixed answers such as a date, an ID or a number; a phrase that must or must not appear; a tool that must be used |
| Python check | Your own code returns a pass or fail, or a score from 0 to 1 | Anything you can count: length, format, a value in a file |
Python checks need your administrator to turn them on. They run in a sandbox when the evaluation runs. The editor can’t try one on a task yet, so run the benchmark to see how it grades.
A task with expected files can be graded only by an agent judge or a Python check, because those are the two kinds that open files. That includes the dataset’s criteria that grade every task.
A monitor can’t use an exact match, because a real request has no expected answer.
Weight what matters
Weights are relative and don’t need to add up to 100. A task’s score is the weighted average of its criteria, so give the criteria that would make an answer unusable, such as an invented fact, more weight than style. A criterion that couldn’t be judged on a task is left out of that task’s score instead of counting as a fail.