AI eval (evaluation)

Output quality drifts when someone edits a prompt or the provider updates the model, and nobody notices until a customer does. An eval catches that drift first, and it gives you evidence when someone asks whether the AI is any good.

How it works

You start with a fixed set of test inputs, each paired with a known good output, called a golden dataset. Graders then score every output. Simple checks run as code, like whether the JSON is valid or a price sits in the allowed range. Quality questions go to a written pass/fail rubric, scored by a person or by a second model acting as a judge.

The first full run sets the baseline, and every later change has to match or beat it before it ships. Scoring a small sample of live outputs each week catches drift between releases.

Example

A marketing ops lead uses AI to write product descriptions for a webshop. She collects 20 past products whose descriptions the brand team approved and writes four pass/fail checks: correct material named, no invented discount, under 120 words, brand tone. Before switching models, she runs all 20 through the old and new versions and compares pass counts.

Common mistake

Starting with a dashboard of generic scores like "helpfulness 4.2 out of 5". The number moves and nobody can say why. Read real outputs first, write down how they actually fail, then build checks for those failures.