Error analysis
Definition
Error analysis is the practice of reading a batch of real AI outputs, noting every way they go wrong, and grouping those failures by type before deciding what to fix or measure. It shows which problems actually happen, so your tests target those.
By Ronan Liedmeier, fractional head of growth and course author · Published 17 October 2026 · Last updated 17 October 2026
Also called: failure analysis
It is the first step in building an eval.
How it works
Pull a sample of recent runs, ideally with their full traces, so you see each step and not only the final answer. Read them one at a time and write a short plain note on anything wrong, like "quoted last year's price" or "tool call failed, report said no data". Then sort the notes into groups and count them.
The biggest groups become your first checks. Each one gets a pass/fail test, written as code where possible and as a judge rubric where it needs reading. The counts also tell you where the fix belongs, since some groups trace back to a prompt and others to a broken data step.
Example
An operator at a recruiting agency runs an AI step that screens CVs against job briefs. She reads 50 screened CVs from last week and tags each problem. Most errors fall into one group: the model treats "3 years of experience" as a hard cutoff when the brief said "around 3 years". A smaller group is CVs laid out in tables that the text extraction step garbled. She writes one judge check for the first and fixes the extraction for the second.
Common mistake
Jumping straight to a dashboard or a ready-made metrics pack. The team ends up tuning metrics that were never tied to how this system actually fails.
Learn it in AI Academy
- The Eval WorkflowExpert · 12 min
Related terms
- AI eval (evaluation)
- An AI eval (evaluation) is a repeatable test that checks whether an AI prompt or workflow gives good answers, by running it on a fixed set of inputs and scoring the outputs against agreed criteria. It turns "seems fine" into a score you can compare.
- Trace
- A trace is the step-by-step record of a single run of an AI workflow or agent, showing every model call and tool call in order with its inputs, outputs, errors and timing. Reading one shows exactly where a run went wrong.
- LLM-as-a-judge
- LLM-as-a-judge is a method of grading AI outputs by asking a second AI model to score them against written criteria, instead of having a person review every answer. It makes quality checks fast enough to run on every change, but the judge itself needs checking.
- Golden dataset
- A golden dataset is a small, fixed set of real example inputs, each paired with an answer you know is right, used to test an AI system. You run every new prompt or model against it and compare the results with those known-good answers.