In short
An evaluation set is a fixed collection of real inputs with defined expectations, run whenever anything changes — prompt, model, retrieval, or provider version. It is the only way to tell an improvement from a regression, and the only defence against providers updating models underneath you. Start with twenty to fifty cases drawn from real usage, including the failures you already know about.
Why this is not optional
With conventional software, a change either passes the tests or does not. With a model in the loop, output varies, correctness is often a judgement, and the provider can change the model without telling you.
That last point is the one teams discover painfully. A prompt tuned over months can start behaving differently with no change on your side. Without an evaluation set, you learn about it from a customer.
Evaluation is also what turns prompt work from taste into engineering. "This feels better" becomes "this scores better on the cases we care about," which is the difference between iterating and wandering.
Build the set from real usage
Not invented examples. Real inputs, from logs if you have them, or from the people who will use the feature.
Twenty to fifty cases is enough to start and is achievable in a day. Include:
Typical cases — the ordinary requests that make up most volume.
Known failures. Every time someone reports a bad output, it becomes a case. This is the highest-value source and it accumulates for free.
Edge cases — empty input, very long input, ambiguous phrasing, mixed languages, adversarial phrasing.
Out-of-scope requests, where the correct behaviour is to decline or redirect. Teams test what the feature should do and rarely test what it should refuse.
Keep it in version control beside the code, so it changes deliberately and reviewably.
What to measure
Depends on the task, and the trick is finding the cheap objective checks before reaching for judgement.
Deterministic checks first. Does the output parse as valid JSON? Do required fields exist? Are types correct? Is it within length? Does it avoid forbidden content? These are free, fast, and catch a surprising share of real failures.
Task-specific correctness. For extraction, compare against known correct values — this is genuinely gradeable. For classification, accuracy against labels. For retrieval-based answers, whether the cited source actually supports the claim, which is the thing that matters most in RAG systems.
Qualities that need judgement — tone, helpfulness, whether an explanation is understandable. These need either a human or a model grading against a rubric.
Grading at scale
Human grading is accurate and does not scale past a few dozen cases per run.
Using a model as a grader scales and is genuinely useful, with caveats worth stating plainly:
Give it a rubric, not a vague instruction. "Rate helpfulness 1–5" produces noise; specific criteria produce something reproducible.
Ask for a binary judgement where you can. "Does this answer contain the required figure — yes or no" is more reliable than a five-point scale.
Validate the grader. Have a human grade thirty cases, compare with the model, and check they broadly agree. An ungraded grader is an assumption.
Use a different model from the one being evaluated where practical, since models tend to rate their own style favourably.
The pragmatic combination: automated checks for everything mechanical, model grading for scale, and human review of a sample each time — particularly the cases the grader marked as passing.
Run it when anything changes
Any prompt edit. Any model version change. Any retrieval or data change. Any parameter change.
And periodically with nothing changed, which is how you detect the provider updating the model underneath you. A scheduled weekly run against a fixed set is cheap and is the only early warning available.
Record the model version, parameters, date and scores with every run. When something moves, you need to know what was different.
Set a threshold and hold it
An evaluation you look at but never act on is a dashboard.
Define what passing means before you need it — no regression on the deterministic checks, and a rate above some level on the judged ones. Then treat a failure as blocking, the way you would a failed test.
Expect some noise, since the same input can produce different output. Run each case a few times if variance matters, and set thresholds with enough margin that ordinary variation does not trip them.
Watching production too
Offline evaluation catches regressions. Production tells you about the inputs you did not anticipate, which is most of them.
Log inputs and outputs, with attention to privacy — this is data you are storing, and it needs the same handling as any other user content.
Track refusal rate, output length distribution, latency, cost per request, and any explicit user signal such as a correction or a retry. A rising retry rate is the most useful early indicator available, because it means people are not getting what they wanted and are trying again.
Then feed the failures back into the evaluation set, which is how it improves.
Starting from nothing
If you have a feature in production with no evaluation, the first useful step is small: collect twenty real inputs, write down what a good answer looks like for each, and run them.
You will immediately learn things — usually that the feature fails in a category nobody had considered. That first run is almost always the most informative one, and it takes an afternoon.
If you have model-backed features in production and no way to tell whether they still work, book a call.
Common questions
Why do LLM features need an evaluation set?
Because output varies, correctness is often a judgement, and providers can update models underneath you without notice. A prompt tuned over months can start behaving differently with no change on your side, and without a fixed evaluation set you find out from a customer.
How many test cases does an LLM evaluation need?
Twenty to fifty is enough to start and achievable in a day. Draw them from real usage rather than inventing them, and include typical cases, every failure anyone has reported, edge cases like empty or very long input, and out-of-scope requests where the correct behaviour is to decline.
Can you use a model to grade another model's output?
Yes, and it is how evaluation scales — but give it a specific rubric rather than a vague instruction, prefer binary judgements over rating scales, and validate it by having a human grade thirty cases and checking they broadly agree. Use a different model where practical, since models favour their own style.
When should an evaluation set be run?
On any prompt, model, retrieval or parameter change — and periodically with nothing changed, which is the only early warning that a provider has updated the model. Record the model version, parameters and date with every run so you can tell what was different when scores move.
What should you monitor in production for an AI feature?
Refusal rate, output length distribution, latency, cost per request, and explicit user signals such as corrections or retries. A rising retry rate is the most useful early indicator, because it means people are not getting what they wanted and are trying again.
