Evals & Review Loops

Lesson preview

The minimum viable eval for any workflow has five parts: a golden set of 20–50 real input→ideal-output pairs (harvested from the manual era of the client's process), deterministic graders for structure, an LLM-as-judge for quality (with a written rubric, forced 1–5 scoring per dimension, and required cited evidence from the output), a baseline score that every change must beat before it ships, and weekly production sampling of 5–10% of live outputs to catch silent drift. Without all five, you have a guess, not a system. A golden set is a collection of 20–50 real input→ideal-output pairs, harvested from the manual era of the client's process — before any AI was involved. Real inputs beat synthetic ones. They surface the edge cases that matter: the weird formatting, the off-topic question, the ambiguous brief that a real user actually sent. Don't treat the golden set as permanent. Update it quarterly with new failure modes discovered through production sampling. Your eval should get harder over time, not easier. Build graders in two layers, from cheapest to most expensive. Deterministic graders first. These check structure: is the output valid JSON? Are the required fields present? Is the length within bounds? Are banned phrases absent? Do the links resolve? Are the dates parseable? These run in milliseconds and catch a huge share of failures. LLM-as-judge second. Use this for quality dimensions that require judgment — tone, completeness, accuracy. But anchor it: write a rubric, force 1–5 scoring per dimension, and require cited evidence from the actual output for each score. Unanchored LLM judges drift optimistic over time. The evidence requirement is what keeps them honest.

Key terms in this module

AI eval (evaluation)

An AI eval (evaluation) is a repeatable test that checks whether an AI prompt or workflow gives good answers, by running it on a fixed set of inputs and scoring the outputs against agreed criteria. It turns "seems fine" into a score you can compare.

Golden dataset

A golden dataset is a small, fixed set of real example inputs, each paired with an answer you know is right, used to test an AI system. You run every new prompt or model against it and compare the results with those known-good answers.

LLM-as-a-judge

LLM-as-a-judge is a method of grading AI outputs by asking a second AI model to score them against written criteria, instead of having a person review every answer. It makes quality checks fast enough to run on every change, but the judge itself needs checking.

Frequently asked questions

Who is "Evals & Review Loops" for?

This Expert module in AI-Native Operator's Field Manual is for operators, founders and growth teams who run AI in real work and want production-grade judgement. It takes about 8 minutes and ends with a 5-question quiz.

Do I need to code?

Not necessarily. The workflows use operator tools such as Claude, n8n and Lovable; some exercises are easier if you are comfortable editing a config file or prompt template.

What comes before and after this module?

Before "Evals & Review Loops", take "Context Engineering" (module 16 of AI-Native Operator's Field Manual). Afterwards, continue with "Guardrails, Permissions & Safety Architecture".

Related modules

• Previous
  Context Engineering

  The defining skill: what does the agent know at the moment of action.

• Next
  Guardrails, Permissions & Safety Architecture

  The moment an agent can spend money, engineering discipline is the product.

• Same topic
  RAG: When It's the Right Answer

  Retrieval and generation are two systems. Evaluate them separately, or debug them never.

• Same topic
  The Eval Workflow

  Error analysis → judges → regression in CI. The skill clients cannot get from a demo.