Skip to module content
🧠AI Academy
0/91
Sign in
Module 03 · ~12 min

The Eval Workflow

Error analysis → judges → regression in CI. The skill clients cannot get from a demo.

Reading progress
0/6 · 0%

The big idea

💡Key idea
The order is the methodology: read real traces first, build cheap judges for the failures you actually observed, then wire it into CI. Teams that skip stage 1 and jump to dashboards full of generic metrics ('helpfulness: 4.2/5') are measuring things that don't reflect what actually breaks in their system.
Quick check
1 question · instant feedback
0/1
  1. Where do the failure categories in a good eval suite come from?

Deep dive

3/3 open

When you read 20–50 real traces from an ads-MCP agent, failures tend to cluster into five buckets. Here they are, ordered by how much damage they do:

**T1 · Wrong tool arguments** — bad date ranges, wrong account IDs, invalid enum values. Frequent, cheap to detect with a code judge.

**T2 · Unfaithful summarisation** — the generated report contradicts the data that was actually retrieved. Moderate frequency, needs a semantic judge.

**T3 · Premature finish** — the agent concludes before checking all relevant campaigns. Only visible at trajectory level, not in the final answer.

**T4 · Error swallowing** — a tool returned an error, but the run reported success anyway. Rare in practice, catastrophic when it happens. This is the one your client remembers: a silenced error in a tool that touches budgets.

**T5 · Over-stepping** — 30 tool calls where 6 would do. Frequent, and it directly affects your cost-per-task number.

Build your judges in severity order. T4 gets a judge before T1 gets a second one.

Think of your regression suite as scar tissue — it encodes every meaningful thing that used to break.

The rule is simple: every time you diagnose and fix a real failure, add the fixture that reproduced it to the golden set. That case now runs on every future PR. This is how regression suites earn their name.

Concretely, a golden case from stage 1 might read: 'Given this account fixture, the report must mention the paused campaign; must call get_ad_performance for a 7-day window; must not exceed 10 tool calls; if the API returns a 429, the run must surface it — not swallow it.'

That's a scenario drawn directly from a real failure. The fixture is the recorded 429 response. The judges are code-checkable. Nothing is left to vibes.

Never assess a change on vibes or a single example you happened to look at.

The discipline is: run the full suite before the change, run it after, and review the specific cases that flipped — pass-to-fail or fail-to-pass. That review takes minutes, not hours, because the judge tells you exactly which behaviour changed.

This is what separates 'ship it, sample some outputs, hope' from a repeatable engineering practice. It's also what turns silent model-provider updates — same model name, different weights — into something you can actually see in your own numbers.

Quick check
1 question · instant feedback
0/1
  1. Best judge shape for 'does this report contradict the retrieved data'?

How to run it

  1. Stage 1 · Error analysis on real traces
    Pull 20–50 traces. Read each end-to-end. For every failure write one plain sentence — not a score. After ~30 traces, cluster the sentences into a failure taxonomy (it will surprise you). Count: frequency × severity tells you what to fix and judge first. Keep the taxonomy open — new failure sentences that fit no cluster start a new cluster.
  2. Stage 2 · Judges
    Code judges first — deterministic, free, never disagree with themselves. Perhaps half a typical taxonomy is code-checkable. LLM-as-judge for the semantic remainder — binary questions with required evidence, never scalar. Validate the judge against ~30 hand-labelled traces; below ~90% agreement, fix the judge prompt. Red-team it: 'construct outputs that fail while passing my judge.' Judge trajectories, not just final answers.
  3. Stage 3 · Regression runner
    Golden set: 30–100 input scenarios with expected behaviours (not exact strings). Fixtures, not live APIs — freeze recorded responses including failures (429s, expired auth, empty results). Runner emits pass/fail per (case × judge) plus cost/latency. Wire to CI. Non-determinism: run flaky cases 3–5× and score pass-rate.
Quick check
1 question · instant feedback
0/1
  1. An agent's final report is correct but it made 40 tool calls (budget was 10). Answer-only evals will…

In the field

🔬Worked example
Your MCP's get_ad_performance handler gets refactored — by an AI, say — and the new version wraps the API call in try/catch that returns an empty result set on failure instead of propagating the error. A rate-limited run now looks like 'no data,' and the downstream report cheerfully says 'no significant spend in the period.' Nothing crashes — this is T4, the silent, catastrophic one. What catches it: (1) a failure fixture — golden case #23 replays a recorded 429; (2) a code judge on the trajectory: tool_error_present && !report.mentions_error → FAIL; (3) CI runs the suite on the PR, case #23 flips from pass to fail. Diff review takes 30 seconds because the judge names the behaviour, not a score.
🚫When not to reach for it
Never skip: even a v0 system deserves error analysis on 20 traces. But scale it honestly — a solo consultant's plain-code harness in the repo is a defensible v1. Adopt a platform when reading JSON traces hurts, not before. And avoid judge scores on a 1–5 Likert scale — they drift, disagree with themselves, and can't be debugged. Binary judgments with required evidence are dramatically more consistent.
Quick check
1 question · instant feedback
0/1
  1. Two golden cases start failing after an AI refactor. You suspect the judge is over-strict. What's the tempting-but-wrong move?

Pitfalls & takeaways

Failure modes

  • Dashboard-first. Generic metrics before error analysis. Numbers move, nobody knows why — sand.
  • Scalar judge scores. 'Rate 1–5' judges drift, disagree with themselves, and can't be debugged. Use binary + evidence.
  • Unvalidated judges. A judge nobody measured against human labels is a random number generator with a UI.
  • Answer-only evals on agents. The report was fine; the run made 40 calls and swallowed two errors. Judge trajectories.
  • Live-API test runs. Flaky, slow, costly, and they mutate real state. Fixtures always.
  • Loosening the judge to make the suite pass. That's recalibrating the instrument to agree with the defect.

Durable takeaways

  • Error analysis first. Judges follow observed failures.
  • Binary judges with required evidence beat scalar scores.
  • Fixtures beat live APIs. Failure fixtures are non-negotiable.
  • Every fixed bug becomes a golden case forever.
Quick check
1 question · instant feedback
0/1
  1. How do you keep a judge trustworthy at scale?

Do the work

🏋️Prove you learned it

Pull 20–50 real traces from your MCP. Read each end-to-end; for every failure write one plain sentence. Cluster into a taxonomy (T1–T5 style). Build the cheapest reliable judge per category (code first, LLM-as-judge for the semantic remainder), validate each judge against 30 hand-labelled traces (>90% agreement). Turn every real failure into a golden case with recorded fixtures. Wire the runner into CI.

0 chars
📦Artifact to produce
Eval harness (fixtures + judges + runner) plus a methodology note explaining the taxonomy.