This page lists every context evals command, what it does and the flag you need most.
Every command except traces get prints one JSON document on standard output (traces get prints a summary and writes the JSON to a file) and takes --help for its full syntax, flags and examples. context evals describe <shape> prints the JSON shape a command reads or writes, with no key needed. Commands marked write need a key with evals:write; the rest need evals:read.
The CLI needs three environment variables: CONTEXT_APP_ORIGIN (your deployment’s address), CONTEXT_ORG_ID (the Organization ID in Settings > General) and CONTEXT_API_KEY. CONTEXT_WORKSPACE_ID is optional and is used when you don’t pass --workspace.
--workflow <id> narrows evals list, evals benchmark list, evals datasets list, evals monitors list and evals traces list to one workflow.
Datasets and tasks
| Command | What it does | Main flag |
|---|
evals datasets list | Lists a workspace’s datasets, a page at a time. | --workspace <id> |
evals datasets get <id> | Shows one dataset and its version state. | |
evals datasets create | Write. Creates a dataset. | --name <text> |
evals datasets update <id> | Write. Changes a dataset’s name or description. | --name <text> |
evals datasets delete <id> | Write. Deletes a dataset. | |
evals datasets duplicate <id> | Write. Copies a dataset and its tasks. | |
evals datasets links <id> | Lists the evaluations that use a dataset. | |
evals datasets import | Write. Imports tasks with their criteria and files from a CSV, JSON or zip file. | --from <file>, --check to preview |
evals datasets versions <id> | Lists a dataset’s saved versions, newest first. | |
evals datasets save <id> | Write. Saves unsaved task and criteria changes as the next version. | --base-version N --working-copy-hash H |
evals datasets discard <id> | Write. Throws away unsaved changes, back to a saved version. | --base-version N --working-copy-hash H --yes |
evals datasets items list <dataset-id> | Lists a dataset’s tasks. | --limit N |
evals datasets items get <item-id> | Shows one task. | |
evals datasets items create | Write. Adds a task. | --dataset <id> --prompt <text> |
evals datasets items update <item-id> | Write. Changes a task. | --golden <text> |
evals datasets items delete <item-id> | Write. Removes a task. | |
evals file upload <path> | Write. Uploads an input or expected file for a task and prints its ID. | --dataset <id> |
Criteria and models
| Command | What it does | Main flag |
|---|
evals criteria get | Shows the criteria on a dataset or a task. | --dataset <id> or --task <id> |
evals criteria set | Write. Replaces a dataset’s or a task’s criteria with a JSON file. | -f <file> |
evals criteria add | Write. Copies a criterion one task carries onto other tasks. | --dataset <id> --criterion <name> --all |
evals criteria remove | Write. Takes a criterion off tasks. | --dataset <id> --criterion <name> --all |
evals criteria library | Lists your organization’s shared criteria, the ones a monitor can bind. | --search <text> |
evals criteria share | Write. Adds or updates one shared criterion, such as an agent judge, and prints its binding. | -f <file> |
evals models list | Lists the models an agent can run, and the judge models. | --agent <id> or --evaluation <id> |
Evaluations and runs
| Command | What it does | Main flag |
|---|
evals benchmark list | Lists a workspace’s evaluations. | --workspace <id> |
evals benchmark get <id> | Shows one evaluation. | |
evals benchmark create | Write. Creates an evaluation from an axis file. | --axes <file> |
evals benchmark update <id> | Write. Changes an evaluation. | --judge-model <model> |
evals benchmark delete <id> | Write. Deletes an evaluation. | |
evals benchmark duplicate <id> | Write. Copies an evaluation and its dataset links, without runs. | |
evals benchmark links <id> | Lists the datasets linked to an evaluation. | |
evals benchmark link | Write. Links a dataset to an evaluation. | --benchmark <id> --dataset <id> |
evals benchmark unlink | Write. Removes a dataset from an evaluation. | --benchmark <id> --dataset <id> |
evals benchmark run create | Write. Starts a run of every task in every cell. | --benchmark <id> |
evals benchmark run list <benchmark-id> | Lists an evaluation’s runs. | |
evals benchmark run get <run-id> | Shows a run’s status and counts. | |
evals benchmark run watch <run-id> | Waits for a run to end and prints its final status. Exits 0, 3 or 4 by outcome; see Errors. | --out <path> |
evals benchmark run stop <run-id> | Write. Asks a run to stop. Exits 5 if it had already ended. | |
evals benchmark run tasks <run-id> | Lists one cell’s tasks with score, verdicts, cost and latency. | --cell <cell-id> |
evals benchmark run task <run-id> <task-id> | Shows one task’s full answer and each criterion’s reasoning. | --cell <cell-id> |
Monitors
These need monitoring, which your administrator turns on. See Monitors.
| Command | What it does | Main flag |
|---|
evals monitors list | Lists a workspace’s monitors. | --agent <id>, --workflow <id> |
evals monitors get <id> | Shows one monitor: targets, criteria, pass mark, state and revision. | --as-update |
evals monitors create | Write. Creates a monitor from a JSON file. | --from <file> |
evals monitors update <id> | Write. Replaces a monitor’s settings from a file, or changes one field. | --from <file> or --pass-mark <n> |
evals monitors pause <id> | Write. Stops grading new requests. Results stay. | --revision <n> |
evals monitors resume <id> | Write. Grades new requests again, charged to you. | --revision <n> |
evals monitors delete <id> | Write. Deletes a monitor and its results. | --revision <n> |
Evaluations of any kind
These treat monitors (monitor:<id>) and evaluations (benchmark:<id>) the same way.
| Command | What it does | Main flag |
|---|
evals list | Lists monitors and evaluations together, with counts for each. | --kind monitor|benchmark |
evals get <ref> | Shows one, with its criteria. | |
evals create | Write. Creates a monitor or an evaluation from a JSON file. | --from <file> |
evals run <ref> | Write. Starts a run of an evaluation. A monitor runs on its own. | |
evals results <ref> | Lists graded results, newest first: request, answer, score, pass and each criterion’s verdict. | --limit N |
evals add-rows | Write. Adds tasks to a dataset from a JSON file. | --dataset <id> --from <file> |
evals targets <kind> | Lists your agents, skills, applets or workflows by name, with the IDs a monitor names. | --search <text> |
Traces, usage and feedback
| Command | What it does | Main flag |
|---|
evals traces list | Lists trace sessions with filters and search. | --from <when> |
evals traces get <session-id> | Writes one session’s full detail to a file. | --out <path> |
evals filter-options | Lists the models, providers, agents and users that have trace data. | |
evals agent <agent-id> | Shows one agent’s cost, tokens and daily usage. | --from <when> |
evals user <user-id> | Shows one user’s cost, tokens and usage by agent. | --from <when> |
evals feedback list | Lists feedback, newest first. | --skill <id> |
Scorecards
| Command | What it does | Main flag |
|---|
evals scorecard create | Write. Creates a rubric and attaches it to a skill. | --skill <id> |
evals scorecard list | Lists the rubrics attached to a skill. | --skill <id> |
evals scorecard get | Shows one rubric. | --scorecard <id> |
evals scorecard set-criteria | Write. Replaces a rubric’s questions. | --from <file> |
evals scorecard rename | Write. Renames a rubric. | --name <text> |
evals scorecard set-description | Write. Sets or clears a rubric’s description. | --description <text> |
evals scorecard attach | Write. Attaches a rubric to a skill. | --scorecard <id> --skill <id> |
evals scorecard detach | Write. Detaches a rubric from a skill. | --scorecard <id> --skill <id> |
Help
| Command | What it does | Main flag |
|---|
evals describe | Prints the JSON shape of any CLI input or output. Needs no key. | |
