Evals API and CLI
Read your organization’s evaluation results and traces over HTTP, and build and run evaluations from the command line.
Evals measures how well your organization’s agents answer. You give it a dataset of tasks, the criteria that grade them and the models or agents to compare. It runs each task, grades each answer and keeps the results. It also keeps a trace of every agent session, so you can find failures and see what your agents’ real work costs.
You can reach Evals in two ways:
| Interface | What it does | Capability needed |
|---|---|---|
HTTP API, under /api/v1/evals | Reads evaluations, datasets, runs, traces, feedback and scorecards. It is read-only: every endpoint is a GET. | evals:read |
context evals | Builds datasets and criteria, creates evaluations, starts and watches runs, and reads results. | evals:read, plus evals:write for commands that change data |
Both use a task API key.
Core concepts
| Concept | What it is |
|---|---|
| Dataset | A named list of tasks in a workspace. A dataset has versions: changes to its tasks and criteria stay unsaved until you save them, and a run grades the latest saved version. |
| Task | One prompt, with an optional expected answer, optional input files the agent reads, and optional expected files that only an agent judge or a Python criterion can open; the agent being tested never sees them. The HTTP API and the CLI call a task an item. |
| Criteria | What grades an answer. A dataset’s criteria grade every task, and a task can carry more of its own. |
| Evaluation | What you run: one or more datasets, plus an axis that names what you compare (models, agents or skill versions). The CLI and the HTTP paths call an evaluation a benchmark. |
| Run | One execution of an evaluation. A run has one cell for each pair of axis value and repeat (a repeat runs the same tasks again, to show how much scores vary): two models with three repeats make six cells. Set repeats with --runs-per-task when you start the run. |
| Trace session | One conversation with an agent, with every model call, its tokens, cost and errors. Traces cover real work, not only evaluation runs. |
| Feedback and scorecards | What people said about an answer (any of a score, a yes or no, and a comment), and the rubrics they fill in when they rate a skill’s answers. |
| Monitor | Grades a sample of real requests as they finish. See Monitors. |
| Workflow | A kind of work your people do, such as refund requests. Chats are filed into workflows so you can filter by them. See Workflows. |
A criterion has one of four kinds:
| Kind | How it grades |
|---|---|
llm_judge | A model reads the answer and your instructions and returns a score from 0 to 1, or a pass or fail. |
agent_judge | An agent that can open files reads the answer and the task’s files, and returns a verdict. It is slower and costs more than llm_judge. A task with expected files can be graded only by an agent judge or a Python criterion. See Agent judge. |
code_verifier | A Python check you write runs on the answer and returns a pass or fail, or a score from 0 to 1. On a monitor it needs a passMark, the score it passes at. On a dataset it needs your administrator to turn Python criteria on. |
exact_match | Checks the answer without a model. It passes when the answer equals the expected answer, contains or doesn’t contain a given text, or when the agent called a named tool. |
Before you start
- The administrator of your deployment (the Context address your organization signs in to) must turn Evals on. Until then, the Evals pages don’t appear in the app.
- Reading most data needs Evals access in the app. Organization owners and admins have it by default, and an admin can grant it to other roles.
- Agent judges, and dataset criteria in general, need the task engine your deployment uses by default. If your administrator switched to the older engine, saving criteria is refused.
- Runs call models, which your organization pays for.
Start with the quickstart.