Errors
The error format, every status code the HTTP API returns and the CLI’s exit codes.
Error format
Errors are JSON with an error message. Some also carry a code you can branch on (see the table), and a 400 carries details. A 429 body has its own shape, shown on Limits; rely on its status and Retry-After.
{ "error": "Invalid request", "details": [ { "path": "pageSize", "message": "Number must be less than or equal to 100" } ]}| Field | Type | Meaning |
|---|---|---|
error | string | A message for people. Don’t parse it; its wording can change. |
code | string | A stable code for programs, when the error has one. |
details | array | On 400, one entry per parameter that failed, with its path and a message. |
Status codes
| Status | code | When | What to do |
|---|---|---|---|
400 | (none) | A parameter fails its schema. | Fix the parameters details names. |
400 | SEARCH_NOT_TOKENIZABLE | A trace search has no word the index can match, such as only punctuation. | Search for words or numbers. |
400 | (none) | On GET /runs/{runId}/tasks or GET /runs/{runId}/tasks/{taskId}, the cellId, cursor or taskId isn’t part of this run, or the run is from before per-task records were kept. | Use values from this run’s own responses. |
400 | INVALID_CURSOR | A trace cursor is malformed or no longer valid, for example after your deployment was upgraded. | Start again without cursor. |
401 | (none) | The request has no task API key, or the key is unknown, revoked or expired. | Send a valid key. See API keys. |
403 | TASK_CAPABILITY_MISSING | The key lacks evals:read. | Create a key with Read evals. |
403 | (none) | The key is limited to specific agents, and the endpoint isn’t a trace read, /feedback/enabled or /feedback/schemas. | Use a key that isn’t limited to agents. |
403 | (none) | The key’s owner lacks the access the endpoint needs. | Ask an admin for Evals access. |
403 | (none) | A list was read without workspaceScope, and the key’s owner has Evals access in more than one workspace but isn’t an organization admin. The message begins with the words Name a workspace. | Send workspaceScope, as in the quickstart. |
404 | (none) | The resource doesn’t exist in your organization. The response is the same whether it doesn’t exist or belongs to another organization. | Check the ID. |
405 | (none) | The method isn’t GET. The Allow header lists the methods. | Use GET. |
429 | (none) | The key exceeded the rate limit, on a deployment that enforces it. | Wait for the seconds in Retry-After, then retry. See Limits. |
500 | QUERY_FAILED | A trace query failed. | Retry with backoff. |
500 | (none) | An unexpected server error. | Retry with backoff. |
503 | (none) | A filter (workflow, error message) couldn’t be applied. | Retry with backoff. |
503 | CLICKHOUSE_UNAVAILABLE | This deployment has no trace store configured, so trace data can’t be read. | Don’t retry; ask your administrator. |
503 | TRACE_READ_SOURCE_UNRESOLVED | The server is still locating this request’s trace data. It clears on its own. | Retry with backoff; if it persists, contact your administrator. |
Codes are case-sensitive; compare them exactly as shown.
A 401 body:
{ "error": "Unauthorized" }A 403 body for a missing capability:
{ "error": "Task API key missing required capability: evals:read", "code": "TASK_CAPABILITY_MISSING"}A 404 body:
{ "error": "Resource not found in this organization" }Retrying
Retry only 429, 500 and 503, but not a 503 with code CLICKHOUSE_UNAVAILABLE, which doesn’t clear on its own. For 429, wait for the Retry-After seconds. For 500 and 503, back off exponentially, for example after 1, 2, 4 and 8 seconds, then stop and report the error. Every HTTP endpoint only reads, so retrying an HTTP request is always safe. A CLI command that changes data is different: check whether the first attempt succeeded before you run it again.
CLI exit codes
On failure the CLI prints Error: <message> on standard error.
| Exit code | Meaning |
|---|---|
0 | Success. For benchmark run watch, the run completed, though some tasks may have scored poorly or errored. |
1 | Failure: a server error (5xx), a rate limit (429), a request too large (413), a network error, or a response the CLI can’t read. A missing environment variable also exits 1, as does benchmark run stop when the runner can’t be reached. Retrying can help, except after a 413 (send less data) or a missing environment variable. For a command that changes data, first check whether the first attempt succeeded. |
2 | Refused: a bad flag, a bad local file, a --pick path that names nothing, or the server answered 400, 401, 403, 404, 409, 412 or 422. benchmark run watch also exits 2 on an unknown or forbidden run, after three tries. Fix the command before retrying. |
3 | benchmark run watch: someone stopped the run. |
4 | benchmark run watch: the run failed. As with every ending, the final run status is on standard output. |
5 | benchmark run stop: the run had already ended, so nothing was stopped. Treat it as success in scripts that only need the run stopped. |
130 | benchmark run watch: you pressed Ctrl-C. The watcher stopped; the run goes on. |
2 means refused for the datasets, benchmark, criteria, monitors, file, models, feedback and evaluation commands; elsewhere any failure is 1. In scripts, treat any non-zero exit as failure.