Monitors

A monitor grades a sample of your agents’ real requests as they finish.

An evaluation runs a fixed dataset when you ask. A monitor works on live traffic instead: it grades a percentage of the real requests to an agent, skill, applet or workflow as each one finishes, with no run to start. The app and the CLI call a monitor a live evaluation.

Monitoring is off until your administrator turns it on for your organization. Until then, monitor commands exit 2. evals monitors list says Monitoring is off for this organization: no live evaluation is listed. The other evals monitors commands, criteria library, criteria share and results monitor:<id> say Benchmark API error 403: Monitoring is not enabled for this organization. Monitors don’t appear in the app.

How a monitor grades

Part	What it does
Targets	What it watches, 1 to 50 of them. Each is an agent, skill, applet or workflow with a sample percentage from 1 to 100, or traces for every request in the organization. Sampling is fixed per request, so the same request is always in or always out.
Criteria	The checks, up to 50: an LLM judge, a Python criterion, or an agent judge from your organization’s shared library (see Agent judge). Each has a weight. An agent judge grades only monitors that target an agent, a skill or traces; on applet and workflow targets it does not run.
Pass mark	A score from 0 to 1. A graded request passes when its weighted score reaches the mark.

A criterion that can’t reach a verdict (for example, its judge ran out of time) reads “Couldn’t be judged” and is left out of the score and the pass rate. A monitor can’t use exact_match, because a live request has no expected answer.

A monitor’s checks are charged to one person: whoever last created, changed or resumed it. A key that has no person behind it can’t do any of those.

Create a monitor

Copy your workspace ID as in the quickstart. Then find the agent’s ID, and the ID of a shared judge to grade with:

context evals targets agent --search refunds --pick 'items[].id'
context evals criteria library --search tone | jq '.items[] | {kind, criterionId, name}'

Save the monitor as monitor.json. This one grades a quarter of one agent’s requests with one shared LLM judge:

{
  "name": "Refund answers",
  "passMark": 0.7,
  "verifiers": [],
  "source": {
    "live": {
      "targets": [{ "kind": "agent", "targetId": "<agent-id>", "samplePct": 25 }],
      "criteria": [{ "kind": "llm_judge", "criterionId": "<criterion-id>", "weight": 1 }]
    }
  }
}

Instead of binding a shared judge in criteria, you can define an LLM judge (or a Python check with a passMark) in verifiers. Give it the fields of a dataset criterion plus an id you choose; it is saved to the shared library. An agent judge must be shared first (see Agent judge). context evals describe evaluation-create-input prints the full shape.

MONITOR_ID=$(context evals monitors create --from monitor.json --workspace "$WORKSPACE_ID" --pick ref.id)

Grading starts with the next request. Running create twice makes two monitors.

Read results and change it

context evals results "monitor:$MONITOR_ID" --limit 20
context evals monitors update "$MONITOR_ID" --pass-mark 0.8
context evals monitors pause "$MONITOR_ID"

Each result row has the request, the answer, its score, whether it passed and each criterion’s verdict. A change names the revision it read, so two people can’t overwrite each other: a change made after someone else’s is refused with exit 2. To change targets or criteria, save monitors get <id> --as-update to a file, edit it, and pass it back with monitors update <id> --from <file>. For a monitor that binds an agent judge, --as-update and the update flags fail with exit 1 and the message A live evaluation's verifier … is a agent_judge, which no criterion binds. Exit 1 means retrying can help, but here it never does, so don’t let a script retry it. --revision does not help update; only update --from <file> works. In the file, list every criterion as {kind, criterionId, weight}, using agent for an agent judge and code for a Python one. Pass --revision to pause and resume. monitors delete needs --revision and removes the monitor’s results too.

Over HTTP, a trace session that a monitor’s agent judge ran in carries origin.monitor with the monitor’s ID and name. Monitors and their results have no HTTP endpoints; use the CLI or the app.