Compare two models from the CLI

This example builds a small dataset, grades it with a model judge and compares two models on one agent.

This example takes an empty workspace to graded results. You need the CLI set up as in the quickstart. You also need a key with both Read evals and Change evals (the quickstart key has only Read evals; create another as on API keys), and jq to read values out of each JSON response. Runs call models, which your organization pays for.

Set the workspace and agent

Copy the workspace ID as in the quickstart. For the agent, context evals filter-options lists the agents that have trace data, with their IDs.
Terminal window
```
export WORKSPACE_ID="<workspace-id>"
export AGENT_ID="<agent-id>"
```

Create a dataset and add tasks

Each task is a prompt and, optionally, the answer you expect (--golden). Run the second command once per task; a few tasks make the comparison mean something.

DATASET_ID=$(context evals datasets create --name "Refund questions" --workspace "$WORKSPACE_ID" | jq -r .id)
context evals datasets items create --dataset "$DATASET_ID" \
  --prompt "A customer bought a laptop 40 days ago. Can they get a refund?" \
  --golden "No. Refunds are available for 30 days after purchase."

Set the criteria

Save this as criteria.json. It asks a model judge whether each answer matches the policy:
```
{
  "definitions": [
    {
      "name": "Matches the refund policy",
      "kind": "llm_judge",
      "weight": 1,
      "outputType": "boolean",
      "instructions": "Pass if the answer agrees with the expected answer: {{expected_output}}. Answer: {{output}}",
      "modelOverride": null
    }
  ]
}
```
outputType is boolean for pass or fail, or float for a score from 0 to 1. In instructions, {{output}} is the agent’s answer and {{expected_output}} is the task’s expected answer. weight sets how much this criterion counts toward the task’s score, relative to the others. modelOverride: null uses the evaluation’s judge model. context evals describe criteria-set prints the file’s full shape.
Terminal window
```
context evals criteria set --dataset "$DATASET_ID" -f criteria.json
```

Save the dataset

Tasks and criteria stay unsaved until you save them, and a run grades only the latest saved version. Save the version you reviewed:

DATASET=$(context evals datasets get "$DATASET_ID")
context evals datasets save "$DATASET_ID" \
  --base-version "$(echo "$DATASET" | jq -r .version.head.version)" \
  --working-copy-hash "$(echo "$DATASET" | jq -r .version.workingCopyHash)"

Create the evaluation

List the models your agent can run, and pick two:

context evals models list --agent "$AGENT_ID" | jq -r '.common[].id'

Save the axis as axes.json, with your agent ID and the two model IDs. testing names what changes between cells, values lists the models to compare, and held fixes everything else:

{
  "testing": "model",
  "values": [
    { "kind": "model", "model": "<provider>/<model-a>" },
    { "kind": "model", "model": "<provider>/<model-b>" }
  ],
  "held": { "agentId": "<agent-id>", "skills": [] }
}

The CLI calls an evaluation a benchmark. --judge-model sets the model that grades llm_judge criteria; pick one with context evals models list --agent "$AGENT_ID" | jq -r '.judges.models[].id'. Without it, the deployment’s default small model grades. Create the evaluation and link the dataset:

EVALUATION_ID=$(context evals benchmark create --name "Refund policy" --axes axes.json --judge-model "<provider>/<judge-model>" --workspace "$WORKSPACE_ID" | jq -r .id)
context evals benchmark link --benchmark "$EVALUATION_ID" --dataset "$DATASET_ID"

Run it and read the scores
Terminal window
```
RUN_ID=$(context evals benchmark run create --benchmark "$EVALUATION_ID" | jq -r .id)
context evals benchmark run watch "$RUN_ID" > status.json
jq '.run.cells as $cells | .stats[] | . as $s | {cellId, cell: ($cells[] | select(.cellId == $s.cellId) | .value), avgScore}' status.json
```
watch prints progress on standard error and the final status on standard output. It exits 0 when the run completed, 3 when someone stopped it and 4 when it failed. It exits 2 when the server refuses (an unknown run ID, say), 130 when you press Ctrl-C and 1 when the watch itself failed (the run may still be going). The last line prints each cell’s ID, model and average score.

To see each task’s answer and verdicts for one cell, use context evals benchmark run tasks "$RUN_ID" --cell <cell-id> with a cellId from that output.

In CI

Install the CLI in the job with the command from Settings > Terminal, set CONTEXT_APP_ORIGIN, CONTEXT_ORG_ID and CONTEXT_API_KEY (as a secret), plus WORKSPACE_ID and AGENT_ID, and run the same commands. Exit 0 from watch means only that the run completed, even if every task scored 0. To fail the job on low scores, check the scores:

jq -e '[.stats[].avgScore] | min >= 0.8' status.json