Dataset file templates

Instead of adding tasks one at a time, you can import them from a file. On a dataset, click Add tasks > Import from a file, choose the file, then pick what the file does to the dataset:

Add to its tasks: the file’s tasks go after the dataset’s own, and its criteria for every task join the dataset’s.
Replace its tasks: the current tasks are archived, so runs that graded them keep their results, and the file’s tasks and criteria for every task take their place.

Then click Import tasks. Once the file is checked, the button counts its tasks, for example Import 12 tasks. You can also import a file when you create a dataset, on New dataset.

The import dialog also offers four templates to start from: Rubric criteria, Expected answers, Files in and out and Blank.

Imported tasks arrive as drafts. A run grades the dataset’s latest saved version, so after an import into an existing dataset, check the tasks and save the dataset with Save as v2 (or whichever version is next).

File types

File	What it holds
`.csv`	One row per criterion, or one row per task when each task has one expected answer. Older `prompt,name,goldenOutput` files still import
`.json`	The whole dataset: settings, criteria and tasks
`.jsonl`	One task per line, with no settings and no criteria for every task. If any line is wrong, the whole file is refused and every bad line is named
`.zip`	A `.csv` or `.json` plus the files its tasks attach

A task’s input files and expected files are paths inside the zip or https:// links. A .csv, .json or .jsonl on its own can only link files. A task can have up to 10 input files and 10 expected files, each up to 10 MiB, and one file can hold up to 1,000 tasks.

How a task is graded

Each task is graded one of two ways:

Expected answer: one right answer, as text, files or both, and the score is how well the answer matches it.
Rubric: weighted criteria the answer must meet.

The dataset’s own criteria (the Shared rubric on a benchmark) grade every task as well, whichever way it is graded. A task with a rubric adds its own criteria on top.

A file can give a task both an expected answer and criteria. It then lands as a rubric, with “Matches the expected answer” as one of its criteria.

A task with expected files can be graded only by an agent judge or a Python check, because only those can open files. That includes the dataset’s criteria for every task.

A task can also be Held out. It’s graded like any other task, but the engine never learns from it, so you can check that an improvement wasn’t fitted to the tasks it came from. You set it on a benchmark’s tasks, not in the file.

Template: expected answers

Use this when each task has a right answer. When a task doesn’t say how to check it (no verifier), an agent judge compares the answer with the expected one and scores it from 0 to 1. The second task uses an exact match instead: the answer must equal the expected text, character for character.

{
  "format": "context-evals-dataset",
  "version": 1,
  "name": "Finance QA (expected answers)",
  "description": "One expected answer per task. Blank verifier = agent judge comparing the output to the expected answer and scoring 0 to 1.",
  "settings": {
    "expectedAnswerVerifier": {
      "kind": "agent_judge",
      "outputType": "float"
    }
  },
  "tasks": [
    {
      "id": "yoy-growth",
      "name": "Year-over-year revenue growth",
      "prompt": "Revenue was $120M in fiscal 2024 and $150M in fiscal 2025. What was the year-over-year growth rate? Show the calculation.",
      "expected": {
        "text": "25% growth: (150 - 120) / 120 = 0.25."
      }
    },
    {
      "id": "q3-end-date",
      "name": "Quarter end date",
      "prompt": "Return only the ISO 8601 date of the last day of calendar Q3 2025. No other text.",
      "expected": {
        "text": "2025-09-30",
        "verifier": {
          "kind": "exact_match"
        }
      }
    }
  ]
}

Template: a rubric

Use this when there’s no single right answer, only things a good answer must do. Each criterion has a weight; a blank weight counts as 1. Criteria in criteria at the top grade every task; criteria inside a task grade only that task. A criterion with no verifier of its own is checked the way defaultVerifier in settings says.

{
  "format": "context-evals-dataset",
  "version": 1,
  "name": "Support and analytics rubric (scorecard)",
  "settings": {
    "defaultVerifier": {
      "kind": "llm_judge",
      "outputType": "boolean",
      "instructions": "You are grading an AI agent's response against one rubric criterion. Pass only if the response clearly satisfies the criterion. Criterion: {{criterion}}. Task: {{prompt}}. Response: {{output}}"
    }
  },
  "criteria": [
    {
      "id": "no-invented-facts",
      "criterion": "Does not state any number or fact that is not in the prompt or attached files.",
      "weight": 20
    }
  ],
  "tasks": [
    {
      "id": "delayed-shipment",
      "name": "Delayed shipment email",
      "prompt": "Draft an email to a customer whose order #4417 will arrive 5 days late because of a carrier delay. The new delivery date is 14 November. Offer free shipping on their next order.",
      "criteria": [
        {
          "id": "apology",
          "criterion": "Apologizes for the delay once and sincerely.",
          "weight": 10
        },
        {
          "id": "new-date",
          "criterion": "States the new delivery date (14 November).",
          "weight": 20
        },
        {
          "id": "offer",
          "criterion": "Mentions the free shipping offer on the next order.",
          "weight": 20
        },
        {
          "id": "length",
          "criterion": "Is 150 words or fewer.",
          "weight": 10,
          "verifier": {
            "kind": "code_verifier",
            "code": "from context_verifiers import Verdict\n\ndef verify(task, output):\n    words = len(output.text.split())\n    return Verdict.boolean(passed=words <= 150, reasoning=f\"{words} words\")\n"
          }
        },
        {
          "id": "tone",
          "criterion": "Uses plain customer-facing language with no internal jargon (no carrier codes or warehouse names).",
          "weight": 10,
          "verifier": {
            "kind": "llm_judge",
            "instructions": "Fail the response if it contains any internal codes, ticket ids or warehouse names. Criterion: {{criterion}}. Response: {{output}}"
          }
        }
      ]
    }
  ]
}

The length criterion is a Python check: it counts words instead of asking a model. Python checks need your administrator to turn them on.

Check a file before you import it

The import dialog checks the file as soon as you choose it, before anything is written. A file that passes shows a preview: each task, whether it’s scored by an expected answer or criteria, how many criteria it has and their total weight. A file with problems lists every one, with the row or line it’s on. Fix them and choose the file again.