Basic Evals for Your Own Workflows

AI-powered workflows don't stay static — prompts get tweaked, the underlying model gets updated, and the kinds of inputs you feed it shift over time. Any one of these can quietly degrade quality without triggering an obvious alarm, because a subjective "it seems fine" impression doesn't reliably catch small, gradual regressions.

Evals exist to replace that vague impression with a number. Without one, you're flying blind — you might not notice a workflow has gotten worse until a client complains, by which point you have no idea which change caused it or how long it's been broken.

A golden set is simply 10 real inputs — not synthetic, not hypothetical — paired with the output you already know is correct or acceptable for each. These become your fixed benchmark: any time you want to know if a workflow still works, you run it against the same 10 inputs and compare.

Building the golden set is itself valuable work, because it forces you to define, concretely, what "correct" looks like for your workflow — something many people have never actually written down, relying instead on an unexamined gut sense.

A useful rubric has 3 to 6 checks, each answerable with a clear yes or no by someone who wasn't involved in building the workflow. "Correct category assigned?" and "draft tone acceptable?" are checkable; "is this good?" is not, because two people could disagree without a shared standard.

Keep the rubric short on purpose — a 20-item checklist becomes tedious to apply consistently and each additional vague item adds noise rather than signal. Three to six sharp, binary checks beat twenty fuzzy ones.

You don't need special tooling to start. A spreadsheet with one row per golden example and one column per rubric check, filled in by hand as you review each output, is a completely legitimate first version of an eval system. Total the checks passed and you have a score out of however many total checks exist.

The manual version also forces you to actually look at every output closely at least once, which itself often surfaces problems you wouldn't have noticed from a passive skim. Don't let the absence of fancy tooling be an excuse to skip evals entirely — the spreadsheet is the whole point at this stage.

Once you're running evals repeatedly, having an AI apply your rubric to each output can save real time — feed it the rubric, the golden example, and the actual output, and ask it to score each check as pass or fail with a one-line reason. This scales the process well beyond what manual grading can sustain.

The catch: an AI grader can itself be wrong or inconsistent, so periodically spot-check a sample of its grades against your own judgment. If the grader and your judgment disagree often, tighten the rubric's wording rather than trusting either blindly — ambiguous rubric language causes both human and AI graders to disagree with themselves over time.

Evals only pay off if you actually look at them regularly. A short, recurring 15-minute review — checking the current score, noting any specific failures, and picking exactly one fix to make before the next review — keeps quality trending upward instead of drifting down unnoticed.

The discipline of "one fix per week" matters as much as the review itself: trying to fix everything at once makes it hard to tell which change actually helped, while one fix at a time lets you attribute improvement (or regression) directly to a specific, known change.

Run your golden set before and after any prompt change, before and after a model switch, and whenever you start feeding the workflow a genuinely new type of input it hasn't seen before. This turns every change into a measured decision rather than a hopeful guess.

A model-switch decision is a good example of evals removing anxiety: run the same golden set on the current model and a candidate model. If both score 9 out of 10 but the candidate is cheaper or faster, you switch with evidence in hand instead of worrying whether quality will silently suffer.