Skip to content
Context docs

Check quality with evals

An evaluation grades the answers of an agent, skill or model. Run one before and after a change to see whether the change helped.

Evals appears only when your organization has it turned on. Organization owners and admins can open it, and so can workspace admins for their own workspaces. Anyone else needs an admin to give them access. If you click Evals on the rail and see “Evals isn’t available on this account”, ask an organization admin or a workspace admin.

What it’s made of

  • A dataset is a named list of tasks. Each task is a prompt for the agent, with optional input files it reads, an optional expected answer, and optional expected files that the agent being tested never sees.
  • A dataset has versions. Your edits stay unsaved until you save a new version, and a run grades the latest saved version.
  • Criteria grade each answer. Together they’re the rubric. A dataset’s criteria grade every task, and a task can add its own. Each criterion is an LLM judge (a model reads the answer), an agent judge (an agent that can open files), a Python check you write, or an exact match (no model). Only an agent judge or a Python check can grade a task with expected files.
  • An evaluation is a benchmark or a monitor. A benchmark runs a dataset when you ask, comparing models, skills or agents side by side. A monitor grades a sample of real requests as they finish, with no run to start.
  • A run is one execution of a benchmark. It tries every task once for each model, skill or agent compared, and again for each repeat you set with Runs per task. Each pair of compared value and repeat is a cell: two models with three repeats make six cells.
  • A workflow is a kind of work your people do, such as refund requests. When your admin turns workflows on, chats are filed into them, so you can narrow evaluations and datasets to one kind of work.

Use real examples, taken from an agent’s Activity tab or from past tasks. With only a handful, you can’t tell a real improvement from chance, so aim for at least 20.

Write the rubric as checks that can be scored. “Cites a source file” and “Under 150 words” work. “Is good” doesn’t.

Run a benchmark

  1. Go to Evals > Datasets > New dataset. Give it a name, such as “Sales product questions”. To start from a file, add it under Import tasks. Click Create dataset.
  2. On the dataset, add tasks with Add task, and the criteria that grade them. Then save a new version with Save as v2 (the number goes up each time), so runs see your edits. If you forget, Create and run offers to save them first.
  3. Go to Evals > Evaluations > New benchmark.
  4. Fill in the page from the top: a name; Compare (Models, Skills or Agents), the ones to compare and what stays the same for every answer; Dataset (click Add dataset and pick yours); and Pass mark.
  5. Click Create and run. The benchmark is saved, its first run starts with each task tried once, and it opens on Activity.

A new benchmark grades with the default judge model. Once it’s saved, you can change that in its Judge model row.

Read the results

On the Activity tab, click a run to see its score and how each task did. Click a task to see each answer and how every criterion scored it, with one tab per cell. The average alone hides what went wrong, so read the tasks that failed.

To see how much scores vary from one try to the next, run again with Runs per task above 1: choose Run with options… in the benchmark’s menu, or click Run again on a run.

To find out whether a change helped, edit one thing, such as the agent’s instructions or a skill, and run the same benchmark again. If you change two things at once, you won’t know which one made the difference. Rerun the benchmark whenever the instructions, skills or model change.

An agent’s or skill’s Evaluations tab lists every evaluation that grades it.

Resources

Use Evals from your own code

Developers can read results over HTTP and build and run evaluations from the command line, with a task API key that can read evals. The Evals developer guide covers both, from a quickstart to the full API and CLI reference. Your deployment also serves the API reference at /api/v1/evals/docs.