Agent judge

An agent judge is a criterion graded by an agent that can open files, for answers a model can’t grade from text alone.

An LLM judge reads the agent’s answer as text. An agent judge is an agent of its own: it opens the files involved, looks through them, and then gives a pass or fail or a score. Use it when the answer is a file, such as a spreadsheet, a report or code, or when grading means comparing against an expected file.

You can use an agent judge in two places: as a criterion on a dataset, which grades evaluation runs, and as a shared criterion that a monitor binds, which grades live requests. Agent judges, and dataset criteria in general, need the task engine your deployment uses by default. If your administrator switched to the older engine, saving criteria is refused. The monitor version also needs monitoring, which your administrator turns on.

Pick a model

An agent judge runs only on models your deployment’s agents can run. List them:

context evals models list --agent "$AGENT_ID" | jq -r '.judges.agentModels[].id'

A model that isn’t on this list is refused when you save, with a message that names it.

On a dataset

Give a task expected files, and add an agent_judge criterion that compares the answer against them. Only an agent judge or a Python criterion can open expected files. While a task has expected files, it and its dataset may carry only those kinds. Set the criteria first (criteria set replaces the whole set), then attach the file. Save this as criteria.json:

{
  "definitions": [
    {
      "name": "Totals match the expected workbook",
      "kind": "agent_judge",
      "weight": 2,
      "outputType": "boolean",
      "instructions": "Open the expected workbook and the workbook the agent made. Pass if every monthly total matches.",
      "modelOverride": "<provider>/<model>"
    }
  ]
}

context evals criteria set --dataset "$DATASET_ID" -f criteria.json

Then add a task, upload the expected file and attach it to the task:

ITEM_ID=$(context evals datasets items create --dataset "$DATASET_ID" --prompt "Build the monthly refund workbook for October." | jq -r .id)
FILE_ID=$(context evals file upload expected.xlsx --dataset "$DATASET_ID" --pick fileId)
context evals datasets items update "$ITEM_ID" --expected-file-id "$FILE_ID"

Expected files are never shown to the agent being tested, and an LLM judge can’t be added while a task has them.

The fields are the same as an llm_judge criterion, and outputType is boolean or float. modelOverride: null uses the evaluation’s judge model, but only if an agent judge can run it; otherwise saving is refused. A dataset you import can carry the same criterion as a Markdown file, with needs_files: true in its front matter.

Each agent judge runs a task of its own, so it is slower than an LLM judge and is charged like any other task. --judge-concurrency on evals benchmark run create sets how many run at once.

On a monitor

A monitor binds an agent judge from your organization’s shared library. An agent judge grades monitors that target an agent, a skill or traces. On applet and workflow targets it does not run. Save it as refund-files.json:

{
  "kind": "agent",
  "name": "Refund letter is complete",
  "instructions": "Open the letter the agent wrote. Pass if it names the order, the amount and the refund date.",
  "model": "<provider>/<model>",
  "outputType": "BOOLEAN",
  "python": false
}

BINDING=$(context evals criteria share -f refund-files.json --pick binding)

--pick binding prints {kind, criterionId, weight}; put it in the criteria list of your monitor file. Running share again with the same name updates the criterion instead of adding another. With outputType: "FLOAT", the judge gives a score from 0 to 1 that passes at passMark (default 0.7). Set python: true to let the judge run Python on a copy of the files; it can never change the originals or reach the network.

The judge works in the task’s own sandbox when the request ends, and has 10 minutes. It grades only tasks on the task engine your deployment uses by default. Its cost is charged to the monitor’s payer.

For a monitor that binds an agent judge, monitors get --as-update and the monitors update flags fail with exit 1; edit it with update --from <file> as the Monitors page describes.

When a judge gives no verdict

A check with no verdict is left out of the score and reads as one of these, among others:

Message	What happened
Couldn’t be judged: the judge filed no verdict in its time	The monitor’s judge ran out of time.
Files changed under the judge, so its verdict is not kept	The files changed while it was grading.
The next message arrived before the judge finished, so this turn was not graded	The person kept the conversation going.
Couldn’t be judged: no agent judge was started for this request	The request wasn’t sampled, the task wasn’t on the default task engine, or the monitor targets an applet or workflow, where an agent judge does not run.