Use Context

Dataset files

A dataset file imports a dataset’s tasks, expected answers and criteria in one step. Import one from a dataset with Add tasks > Import from a file.

Import a dataset file

1. On a dataset, click Add tasks > Import from a file and choose the file. To import while you create a dataset, add the file on New dataset instead.
2. Pick what the file does to the dataset:
   • Add to its tasks: the file’s tasks go after the dataset’s own, and its criteria for every task join the dataset’s.
   • Replace its tasks: the current tasks are archived, so runs that graded them keep their results, and the file’s tasks and criteria for every task take their place.
3. Click Import tasks. Once the file is checked, the button counts its tasks, for example Import 12 tasks. Imported tasks arrive as drafts.
4. After an import into an existing dataset, check the tasks and save the dataset with Save as v2 (or whichever version is next). A run grades the dataset’s latest saved version.

The import dialog also offers four templates to start from: Rubric criteria, Expected answers, Files in and out and Blank.

File types

A dataset task’s input files and expected files are paths inside the zip or https:// links. A .csv, .json or .jsonl on its own can only link files. A task can have up to 10 input files and 10 expected files, each up to 10 MiB, and one file can hold up to 1,000 tasks.

A file can give a task an expected answer, criteria or both. A task given both lands as a rubric, with “Matches the expected answer” as one of its criteria. How a task is graded covers both ways of grading.

Validation

The import dialog checks the file as soon as you choose it, before anything is written. A file that passes shows a preview: each task, whether it’s scored by an expected answer or criteria, how many criteria it has and their total weight. A file with problems lists every one, with the row or line it’s on. Fix them and choose the file again.

Expected answers template

Use this when each task has a right answer. When a task doesn’t say how to check it (no verifier), an agent judge compares the answer with the expected one and scores it from 0 to 1. The second task uses an exact match instead: the answer must equal the expected text, character for character.

Code
{
  "format": "context-evals-dataset",
  "version": 1,
  "name": "Finance QA (expected answers)",
  "description": "One expected answer per task. Blank verifier = agent judge comparing the output to the expected answer and scoring 0 to 1.",
  "settings": {
    "expectedAnswerVerifier": {
      "kind": "agent_judge",
      "outputType": "float"
    }
  },
  "tasks": [
    {
      "id": "yoy-growth",
      "name": "Year-over-year revenue growth",
      "prompt": "Revenue was $120M in fiscal 2024 and $150M in fiscal 2025. What was the year-over-year growth rate? Show the calculation.",
      "expected": {
        "text": "25% growth: (150 - 120) / 120 = 0.25."
      }
    },
    {
      "id": "q3-end-date",
      "name": "Quarter end date",
      "prompt": "Return only the ISO 8601 date of the last day of calendar Q3 2025. No other text.",
      "expected": {
        "text": "2025-09-30",
        "verifier": {
          "kind": "exact_match"
        }
      }
    }
  ]
}

Rubric template

Use this when there’s no single right answer, only things a good answer must do. Each criterion has a weight; a blank weight counts as 1. Criteria in criteria at the top grade every task; criteria inside a task grade only that task. A criterion with no verifier of its own is checked the way defaultVerifier in settings says.

Code
{
  "format": "context-evals-dataset",
  "version": 1,
  "name": "Support and analytics rubric (scorecard)",
  "settings": {
    "defaultVerifier": {
      "kind": "llm_judge",
      "outputType": "boolean",
      "instructions": "You are grading an AI agent's response against one rubric criterion. Pass only if the response clearly satisfies the criterion. Criterion: {{criterion}}. Task: {{prompt}}. Response: {{output}}"
    }
  },
  "criteria": [
    {
      "id": "no-invented-facts",
      "criterion": "Does not state any number or fact that is not in the prompt or attached files.",
      "weight": 20
    }
  ],
  "tasks": [
    {
      "id": "delayed-shipment",
      "name": "Delayed shipment email",
      "prompt": "Draft an email to a customer whose order #4417 will arrive 5 days late because of a carrier delay. The new delivery date is 14 November. Offer free shipping on their next order.",
      "criteria": [
        {
          "id": "apology",
          "criterion": "Apologizes for the delay once and sincerely.",
          "weight": 10
        },
        {
          "id": "new-date",
          "criterion": "States the new delivery date (14 November).",
          "weight": 20
        },
        {
          "id": "offer",
          "criterion": "Mentions the free shipping offer on the next order.",
          "weight": 20
        },
        {
          "id": "length",
          "criterion": "Is 150 words or fewer.",
          "weight": 10,
          "verifier": {
            "kind": "code_verifier",
            "code": "from context_verifiers import Verdict\n\ndef verify(task, output):\n    words = len(output.text.split())\n    return Verdict.boolean(passed=words <= 150, reasoning=f\"{words} words\")\n"
          }
        },
        {
          "id": "tone",
          "criterion": "Uses plain customer-facing language with no internal jargon (no carrier codes or warehouse names).",
          "weight": 10,
          "verifier": {
            "kind": "llm_judge",
            "instructions": "Fail the response if it contains any internal codes, ticket ids or warehouse names. Criterion: {{criterion}}. Response: {{output}}"
          }
        }
      ]
    }
  ]
}

The length criterion is a Python check: it counts words instead of asking a model. Python checks need your administrator to turn them on.

Next steps

• Criteria: write criteria that grade the same way every time.
• Benchmarks and monitors: create a benchmark that runs the dataset.
• Read results: find out what a finished run’s score means.