LLM-as-a-judge
Definition
LLM-as-a-judge is a method of grading AI outputs by asking a second AI model to score them against written criteria, instead of having a person review every answer. It makes quality checks fast enough to run on every change, but the judge itself needs checking.
By Ronan Liedmeier, fractional head of growth and course author · Published 17 October 2026 · Last updated 17 October 2026
Also called: LLM judge, AI judge, model-graded eval
Some quality questions can't be checked with code, like whether a reply answers what the customer asked or whether a summary drops the key risk. A judge model can answer them for every output in a test run.
How it works
The judge reads each input and output against a rubric and returns a verdict plus evidence: the exact sentence from the output that justifies it. Narrow pass/fail questions ("does the reply promise a refund the policy doesn't allow?") give steadier results than open 1 to 5 ratings.
Zheng et al. (2023) studied the method and found GPT-4 as a judge agreed with human preferences over 80% of the time, about as often as humans agreed with each other. The same paper documented judge biases, including position bias (favoring the answer shown first) and verbosity bias (favoring longer answers).
Example
A content team uses AI to draft LinkedIn posts from client case notes. A judge model checks each draft against two questions: does it contain a number that is missing from the case notes, and does it name the client. Each fail comes with the quoted sentence, so an editor confirms it in seconds.
Common mistake
Using a judge nobody has checked. A vague prompt like "rate the quality" tends to grade generously. Compare its grades with a person's every few weeks and tighten the rubric where they disagree, instead of loosening it to make a failing test pass.
Learn it in AI Academy
- The Eval WorkflowExpert · 12 min
- Evals & Review LoopsExpert · 8 min
Related terms
- AI eval (evaluation)
- An AI eval (evaluation) is a repeatable test that checks whether an AI prompt or workflow gives good answers, by running it on a fixed set of inputs and scoring the outputs against agreed criteria. It turns "seems fine" into a score you can compare.
- Golden dataset
- A golden dataset is a small, fixed set of real example inputs, each paired with an answer you know is right, used to test an AI system. You run every new prompt or model against it and compare the results with those known-good answers.
- Error analysis
- Error analysis is the practice of reading a batch of real AI outputs, noting every way they go wrong, and grouping those failures by type before deciding what to fix or measure. It shows which problems actually happen, so your tests target those.
- Large language model (LLM)
- A large language model (LLM) is an AI system trained on huge amounts of text to predict which words should come next. At scale, that one skill lets it answer questions, summarize, translate and write in a way that reads like a person wrote it.