Golden dataset
Definition
A golden dataset is a small, fixed set of real example inputs, each paired with an answer you know is right, used to test an AI system. You run every new prompt or model against it and compare the results with those known-good answers.
By Ronan Liedmeier, fractional head of growth and course author · Published 17 October 2026 · Last updated 17 October 2026
Also called: golden set, golden examples
It is the base of any eval: without known-good answers, a score has nothing to compare against. A first version can be as small as ten cases.
How it works
Pull inputs from real work the team already handled by hand, such as past tickets or briefs, and keep the output a person approved. Add awkward cases on purpose, like a ticket written in another language or the input that broke the workflow last month. Each time a bug is fixed, its input joins the set, so the same failure gets caught if it ever comes back.
Keep the set frozen while you compare versions. If the test cases and the prompt both change at once, the two scores can't be compared.
Example
A customer support lead at a subscription box company picks ten past emails: four routine delivery questions, three refund requests, two complaints in German, and one where an agent wrongly promised a refund last spring. Next to each she pastes the reply a senior agent approved. Every prompt change runs against those ten before it goes live, and she checks which emails changed result.
Common mistake
Filling the set with easy cases the system already gets right. A set that always scores ten out of ten tells you nothing about a change.
Learn it in AI Academy
- Basic Evals for Your Own WorkflowsAdvanced · 14 min
- Evals & Review LoopsExpert · 8 min
Related terms
- AI eval (evaluation)
- An AI eval (evaluation) is a repeatable test that checks whether an AI prompt or workflow gives good answers, by running it on a fixed set of inputs and scoring the outputs against agreed criteria. It turns "seems fine" into a score you can compare.
- Regression testing for AI
- Regression testing for AI is re-running a fixed set of test cases after any change to a prompt or model, to check that what worked before still works. It catches changes that raise the average score while breaking one specific case.
- Error analysis
- Error analysis is the practice of reading a batch of real AI outputs, noting every way they go wrong, and grouping those failures by type before deciding what to fix or measure. It shows which problems actually happen, so your tests target those.
- LLM-as-a-judge
- LLM-as-a-judge is a method of grading AI outputs by asking a second AI model to score them against written criteria, instead of having a person review every answer. It makes quality checks fast enough to run on every change, but the judge itself needs checking.