AI evaluation: how to know whether it is working

Someone changes a prompt. They try it on two or three examples, the answers look better, and the change ships. Three weeks later a customer reports that something which used to work no longer does, and nobody can say whether that change caused it, because there is no record of how the system behaved before.

The business consequence is not the individual regression. It is that every subsequent change becomes a guess. Without a way to compare before and after, improvement and regression are indistinguishable, so the team either stops changing anything — freezing a system that could get better — or keeps changing it on the basis of whoever last looked at the output. Confidence in the feature decays either way, and the decay is usually attributed to the model rather than to the absence of measurement.

Evaluation is what replaces testing when the component is not deterministic. It is a measurement discipline rather than a test suite, and it is the single practice that separates AI features that improve over time from ones that are shipped once and quietly distrusted.

Why testing does not apply

Conventional software testing rests on determinism: given the same input, a function returns the same output, so a test asserts an expected value and any deviation is a failure. That assumption is what makes a test suite a reliable gate.

A language model does not offer it. The same prompt can produce different wording, different structure and occasionally different substance. An assertion that the output equals a fixed string will fail on a perfectly good answer, and loosening it until it passes produces a test that also passes on a bad one. There is no threshold at which exact-match assertions become useful for open-ended text.

What replaces them is measurement across a population rather than assertion on a single case. You do not ask whether this output is exactly right; you ask what proportion of a representative set is acceptable, and whether that proportion moved when you changed something. NIST frames this as one of the four functions in its AI Risk Management Framework — Govern, Map, Measure and Manage — and the framing is useful precisely because Measure sits alongside the others rather than being a phase that completes.

What an evaluation actually is

Stripped of tooling, an evaluation has two parts, and OpenAI's own documentation describes them plainly. It defines evals as tests of model outputs to ensure they meet style and content criteria that you specify, and states that an evaluation requires a data source configuration — a schema for the test data you will use — and testing criteria, which are the graders that determine if the model output is correct.

That is the whole concept. A set of cases, and a way to decide whether each answer is acceptable. Everything else is implementation.

One detail in the same documentation is worth pausing on, because it is where the cost actually sits. The guidance for building a test dataset specifies that each item should include the test input and a ground truth label for comparison, provided by a human. The eval set is a human-labelled dataset. It is not generated, it is not free, and it does not maintain itself.

Building the set is the work

Most evaluation programmes stall here, and the stall is usually mistaken for a tooling problem. It is not. Assembling cases and agreeing what a good answer looks like is judgement work involving people outside engineering, which makes it slower to schedule than anything technical.

Take cases from production, not imagination

Cases invented by the person building the feature encode the same assumptions the feature encodes, so they pass. The valuable cases are the ones that already went wrong: the question the assistant answered confidently and incorrectly, the extraction that silently dropped a field, the summary that inverted a meaning. Every complaint about an AI feature is a candidate case with the failure already identified.

This is why an evaluation set built after three months of production use is far more valuable than one built before launch, and why the practice of routing every reported failure into the set is worth more than any single decision about tooling.

Coverage matters more than volume

A set of two hundred cases that all resemble each other measures one thing two hundred times. A set that spans the common request, the ambiguous request, the request that should be refused, the edge case with missing data, and the three specific failures customers have complained about, measures the system. Aim for spread across the categories that matter, then add volume within each.

Someone has to own it

An eval set decays. The product changes, the policy changes, and cases that were correctly labelled last year are now labelled wrong — at which point the evaluation actively misleads, reporting regressions that are actually the system correctly following new rules. A named owner and a scheduled review is the difference between an asset and a source of false alarms.

How to grade an answer

Grading is where cost and reliability trade off against each other. Four approaches cover almost everything, and most serious evaluations use more than one.

The second row is consistently under-used. A large share of what goes wrong with AI features in production is structural rather than semantic — malformed output, a missing field, a value outside the permitted set — and those failures are catchable deterministically, cheaply, on every single request rather than on a sample. Structural checks are not evaluation exactly, but they remove a category of problem before evaluation has to consider it.

The judge has documented biases

Using one model to grade another is now the default technique for scaling evaluation, and it is a reasonable one. It is also worth knowing that the research which established the technique named its failure modes at the same time, and those failure modes are rarely mentioned by the tools that sell it.

The reference work is Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, by Zheng and colleagues, presented at the NeurIPS 2023 Datasets and Benchmarks Track. Its stated motivation is that evaluating chat assistants is challenging because of their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. Its finding on usefulness is genuinely positive: strong judges such as GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement — the same level of agreement between humans.

That number is worth reading in both directions, because it is the most useful single fact in this article. It means an automated judge is about as reliable as a human reviewer, which is a strong endorsement. It also means roughly one judgement in five disagrees, and that this is the floor rather than a defect to be tuned away — humans disagree with each other at the same rate. Any measured improvement smaller than that disagreement is not a signal.

The paper identifies four specific biases by name.

None of these makes the technique unusable. All of them mean an automated judge should be calibrated against human scores on a sample before its numbers are trusted, and re-calibrated when the model behind it changes. A judge whose agreement with your own reviewers has never been measured is producing numbers of unknown meaning.

The arithmetic nobody does

This is the most common way evaluation goes wrong in practice, and it has nothing to do with AI. It is sample size.

A team assembles thirty cases. The current prompt scores 24 out of 30, or 80%. Someone changes it, the new version scores 26 out of 30, and the change ships as an improvement. What actually happened is that two more cases passed. On a set that size, that difference is comfortably within the range you would expect from re-running the same prompt twice against a non-deterministic model, before considering whether the thirty cases represent the workload at all.

The practical consequences are worth stating plainly rather than deriving.

• Small sets detect only large differences. A set of thirty tells you whether something is broadly better or broadly worse. It cannot resolve a few percentage points, and treating it as if it can generates confident nonsense.
• Run the same evaluation twice before believing any change. If the score moves when nothing changed, that movement is your noise floor, and any real improvement has to exceed it.
• Report the count, not just the percentage. Twenty-six of thirty is harder to over-interpret than 86.7%, which is the point.
• Differences smaller than judge disagreement are not measurable at all. If your judge agrees with humans four times in five, a three-point difference scored by that judge is inside its own error.
• Grow the set where decisions are being made. If a category is driving product decisions, that category needs enough cases to support them, which usually means more than the whole set contains today.

This is not an argument for enormous evaluation sets. It is an argument for matching the claim to the evidence. A thirty-case set is genuinely useful for catching a change that broke something badly, which is the most common real failure. It is not useful for choosing between two prompts that are both roughly fine, and most teams use it for exactly that.

Three different things, often confused

Evaluation, guardrails and production monitoring get discussed as one topic and answer different questions at different times.

The third column is the one most often skipped and the one that catches the failure nobody anticipated. An offline evaluation only knows about cases someone thought of; production traffic contains the ones they did not. Sampling real outputs and scoring them on the same rubric is how new failure categories are discovered, and each one discovered becomes a case in the offline set.

Where an estate already runs monitoring dashboards, evaluation scores belong on them alongside everything else, for the same reason integration metrics do: whoever responds to a degradation should not need a second place to look.

When correctness is not binary

Classification and extraction have right answers. Summaries, drafts and explanations do not, and the instinct is to conclude they cannot be evaluated. They can, but the target changes from correctness to conformance with a stated standard.

That standard is a rubric: an explicit list of what an acceptable answer must and must not do. It cites a source. It does not invent a figure. It answers the question asked rather than an adjacent one. It stays within a stated length. Writing the rubric is most of the intellectual work, and it has a useful side effect — a team that cannot agree a rubric has discovered that it does not agree what the feature is for, which is worth finding out before rather than after launch.

Where a human already reviews output before it is used, there is a cheaper measure available. The rate at which reviewers accept the output unchanged is a direct, continuously generated signal of quality that costs nothing extra to collect, because the review is happening anyway. The support-focused article in this series describes that pattern in an operational setting.

And if nobody can describe what a good output looks like even in a rubric, the honest conclusion is the one stated in the AI integration article: an unevaluable system cannot be improved or trusted, and that is a reason to reconsider the use case rather than a gap to work around.

Fitting evaluation into how changes get made

An evaluation that exists but is not run at the moment of change is documentation. Three practices make it operational.

1. 1

   Define what counts as a change

   The prompt, the model or model version, the retrieval configuration, the source content, and any tool the model can call. All five alter behaviour, and teams routinely run evaluation on the first while changing the others silently.

2. 2

   Version the set with the code that uses it

   Cases, rubrics and expected labels belong in the repository and go through review like anything else. An eval set that lives in a spreadsheet on someone's drive cannot be tied to the version of the system it was measuring.

3. 3

   Record the score with the change

   The valuable artefact is the history: what the score was before, what it was after, and how large the set was. Without it, the question of when a regression was introduced has no answer, which is the situation this article opened with.

When not to build an evaluation programme

Evaluation has real cost, most of it in human labelling, and there are cases where the honest answer is not to build one.

Where a person already reviews every output before it is used and volume is low, the review is the evaluation. Collect the acceptance rate, keep a handful of cases for when something changes, and stop there. Where the feature is genuinely low stakes and a wrong answer costs a few seconds of mild irritation, a formal programme is disproportionate. And where the system is a prototype that may not survive the quarter, building a labelled dataset for it is investment ahead of a decision that has not been made.

The threshold worth watching for is the point at which the feature becomes something the business relies on without a person checking each output. That is when the absence of measurement stops being a reasonable economy.

Common mistakes

A representative scenario

A composite of situations we see repeatedly; no client detail is included.

A company runs an assistant that drafts responses from its own documentation. Quality is assessed by whoever is looking at it. A prompt change is made to reduce verbosity, spot-checked on a few examples, and shipped. Two weeks later a support lead notices that answers have stopped citing sources — a behaviour nobody was tracking, removed by a change nobody connected to it.

The set built afterwards came almost entirely from history: forty cases drawn from complaints and known failures, each labelled with what an acceptable answer must contain. Grading combined a programmatic check that a citation was present at all, which was cheap and would have caught this specific regression on the day it shipped, with a rubric-based judge for the qualitative part. The judge was calibrated against two reviewers on a sample before its scores were used.

The first useful discovery was not a score. Running the unchanged system twice produced different results on three of the forty cases, which established that anything smaller than that difference was not measurable with the set as it stood — and stopped a subsequent round of prompt changes from being reported as improvements.

Implementation checklist

• Cases come from production failures and complaints rather than from examples written alongside the feature.
• The set spans common requests, ambiguous ones, ones that should be refused, edge cases with missing data, and every failure customers have reported.
• Each case carries a human-provided view of what an acceptable answer must contain.
• Structural checks run on every request where output feeds another system: valid shape, required fields, values in range.
• Any model-as-judge has been calibrated against human scores on a sample, and is re-calibrated when the judge model changes.
• The judge is not the same model that produced the output.
• Comparisons are run with positions swapped, and length is controlled for or watched.
• The unchanged system has been run twice to establish the noise floor, and no difference smaller than that is reported as a result.
• Scores are reported as counts alongside percentages.
• Evaluation runs on changes to the prompt, the model version, the retrieval configuration, the source content, and available tools.
• Cases, rubrics and labels are version controlled with the code they measure.
• A sample of production outputs is scored on the same rubric, and new failure categories are added to the set.
• A named person owns the set and reviews the labels on a schedule.

Frequently asked questions

Conclusion

Evaluation is unglamorous and it is what separates an AI feature that improves from one that is shipped once and then quietly worked around. The components are modest: cases taken from real failures, a human view of what a good answer contains, cheap structural checks on everything that feeds another system, a judge that has been calibrated rather than assumed, and enough cases that a measured difference means something.

The discipline that matters most is the least technical. Report what you actually measured — the count, the size of the set, and the noise floor you established by running it unchanged — rather than a percentage that sounds like progress. A team that knows how much of its own measurement is noise will make better decisions than one with a larger set and more confidence than it has earned.