Regression testing for AI
Definition
Regression testing for AI is re-running a fixed set of test cases after any change to a prompt or model, to check that what worked before still works. It catches changes that raise the average score while breaking one specific case.
By Ronan Liedmeier, fractional head of growth and course author · Published 17 October 2026 · Last updated 17 October 2026
Also called: AI regression test, regression suite
Changes come from two directions: your own edits, and the provider updating or retiring the model or API underneath you. Either one can break a workflow without a single error message.
How it works
Keep a golden dataset and record which cases pass today. After a change, run the full set again and compare each case with its last result.
Every bug you fix becomes a new case, so the suite grows with each failure the team has seen. When a provider announces a model deprecation, run the suite against the replacement before moving any traffic. Hooking the run into your release process makes it automatic, so nobody has to remember.
Example
A finance operations analyst has an AI step that reads invoice totals and due dates from supplier PDFs. A newer model scores better on her 25 test invoices overall. The case-by-case comparison shows two invoices from one Dutch supplier now come back with day and month swapped. She keeps the old model pinned until a date-format check is in place.
Common mistake
Reading only the headline pass rate. Ten passes where there used to be nine looks like progress, even when the change fixed two cases and broke the one that mattered most, like the invoice that now posts to the wrong month.
Learn it in AI Academy
- Basic Evals for Your Own WorkflowsAdvanced · 14 min
- Building on Moving APIsAdvanced · 12 min
Related terms
- Golden dataset
- A golden dataset is a small, fixed set of real example inputs, each paired with an answer you know is right, used to test an AI system. You run every new prompt or model against it and compare the results with those known-good answers.
- AI eval (evaluation)
- An AI eval (evaluation) is a repeatable test that checks whether an AI prompt or workflow gives good answers, by running it on a fixed set of inputs and scoring the outputs against agreed criteria. It turns "seems fine" into a score you can compare.
- Model deprecation
- Model deprecation is when an AI provider announces that a model or API will be retired and sets a date after which requests to it stop working. Until then it still runs, but users are expected to move to a recommended replacement.
- Model pinning
- Model pinning is the practice of naming one exact, fixed model version in your code or settings, instead of a general name the provider can point at a newer model. It stops the model from changing under you before you have tested the new one.