The Eval Workflow

Lesson preview

The order is the methodology: read real traces first, build cheap judges for the failures you actually observed, then wire it into CI. 2/5') are measuring things that don't reflect what actually breaks in their system. When you read 20–50 real traces from an ads-MCP agent, failures tend to cluster into five buckets. Here they are, ordered by how much damage they do: T1 · Wrong tool arguments — bad date ranges, wrong account IDs, invalid enum values. Frequent, cheap to detect with a code judge. T2 · Unfaithful summarisation — the generated report contradicts the data that was actually retrieved. Moderate frequency, needs a semantic judge. T3 · Premature finish — the agent concludes before checking all relevant campaigns. Only visible at trajectory level, not in the final answer. T4 · Error swallowing — a tool returned an error, but the run reported success anyway. Rare in practice, catastrophic when it happens. This is the one your client remembers: a silenced error in a tool that touches budgets. T5 · Over-stepping — 30 tool calls where 6 would do. Frequent, and it directly affects your cost-per-task number. Build your judges in severity order. T4 gets a judge before T1 gets a second one.

Key terms in this module

AI eval (evaluation)

An AI eval (evaluation) is a repeatable test that checks whether an AI prompt or workflow gives good answers, by running it on a fixed set of inputs and scoring the outputs against agreed criteria. It turns "seems fine" into a score you can compare.

Error analysis

Error analysis is the practice of reading a batch of real AI outputs, noting every way they go wrong, and grouping those failures by type before deciding what to fix or measure. It shows which problems actually happen, so your tests target those.

LLM-as-a-judge

LLM-as-a-judge is a method of grading AI outputs by asking a second AI model to score them against written criteria, instead of having a person review every answer. It makes quality checks fast enough to run on every change, but the judge itself needs checking.

Frequently asked questions

Who is "The Eval Workflow" for?

This Expert module in AI Delivery Skills & Workflows is for operators, founders and growth teams who run AI in real work and want production-grade judgement. It takes about 12 minutes and ends with a 5-question quiz.

Do I need to code?

Not to follow the lesson. The concepts are explained in plain language; if you build AI systems yourself, the exercise shows where the ideas map onto code.

What comes before and after this module?

Before "The Eval Workflow", take "RAG: When It's the Right Answer" (module 2 of AI Delivery Skills & Workflows). Afterwards, continue with "Observability & Tracing".

Related modules

• Previous
  RAG: When It's the Right Answer

  Retrieval and generation are two systems. Evaluate them separately, or debug them never.

• Next
  Observability & Tracing

  First in your build order, even though it's fourth on the page — error analysis needs traces to read.

• Same topic
  Basic Evals for Your Own Workflows

  "Seems good" is not a metric. Ten golden examples are.

• Same topic
  Prompt Engineering at Production Quality

  Prompts as versioned, schema-validated software, not vibes.