How to Measure the Quality of an AI System: A Production Evaluation Framework

Ask a team whether their AI feature is any good and you will get a number. Ninety-one percent accuracy. Four point two out of five. A thumbs-up rate. The number is always specific, always confident, and almost always answering a question nobody asked. The problem is not that the number is wrong. It is that quality is not a scalar, and a scalar cannot tell you which part of the system moved.

This article is about building a measurement system for a production AI feature: what to measure, how to measure it honestly, and where measurement stops being worth the effort. It assumes you have something shipped or about to ship — a retrieval pipeline, an agent, a classifier, an LLM-backed feature. It is not about leaderboard scores or academic benchmarks, though it borrows their techniques where they help.

The through-line is simple. You do not have a quality problem. You have a set of quality dimensions that fail independently, a set of instruments that are each blind to something, and a statistical environment in which the same input does not produce the same output twice. Get those three things right and the numbers become useful. Get them wrong and you will optimize a metric straight into a worse product.

Quality is a vector, not a number

The first discipline is to refuse the single number. A production AI system has several dimensions, and the ones that matter move independently of each other. The ones I have had to instrument:

• Task success — did the system do the thing the user needed, end to end.
• Groundedness — is every claim supported by the evidence the system was given, or by the world as it actually is.
• Policy adherence — did it stay inside the rules of the domain: refund limits, data-handling constraints, tone, legal boundaries.
• Safety — does it refuse or escalate where it should, and never cross a hard line: disallowed content, instructions that cause harm, exposure of data it was not meant to surface.
• Robustness — does it hold on the tail: rare inputs, adversarial inputs, distribution shift, malformed tool responses.
• Cost — not per call, but per successful task. A cheap model that fails and retries is expensive.
• Latency — p50 and p95, and where in the pipeline the time goes.
• Reliability — the variance of all of the above across repeated attempts. Easy to forget, expensive to ignore.

Read that list and notice how easily one dimension moves without the others. A prompt edit that raises policy adherence can flatten the answer's usefulness. A model upgrade that improves task success can double the cost per successful task. A new embedding model can improve offline retrieval recall while the production index quietly drifts out of date and groundedness falls.

A single composite score hides all of it. Track one number and a real regression in one dimension gets cancelled by a cosmetic gain in another, while the dashboard stays green and users churn. The number your dashboard needs is not a number. It is a vector, reported per dimension, with a variance attached.

Capability is not reliability

There is a second collapse that happens before the first one: conflating the system can with the system does. Benchmarks and demo evals measure capability — the best case on a favorable sample. Production is governed by reliability — the expected case on the real distribution, repeated thousands of times.

The difference is not small. Suppose an agent succeeds on a given task 60 percent of the time. A single attempt at 60 percent looks like a coin flip you might ship. Now ask the question a user actually asks: will it work all five times they need it this week. If the attempts were independent, the probability of five consecutive successes is 0.6^5, just under 8 percent. Eight consecutive successes is 0.6^8, about 1.7 percent. Reliability does not add; it multiplies. And the independence assumption is generous — real failures correlate (a rare input type that breaks today breaks the same way tomorrow), which makes the compounding worse, not better.

This is why the code-generation literature reports pass@k — the probability that at least one of k samples passes the tests — while agent evaluation has moved toward something closer to the opposite. The τ-bench benchmark from Sierra introduced pass^k: the probability that all k independent attempts succeed. Same letters, opposite question, and pass^k is the one that maps to "can I deploy this." A system can look excellent at pass@1 and humbling at pass^8.

The evaluation stack

With the dimensions named, you need instruments. They stack in three layers, each cheaper to run and less faithful than the one above it. The order matters, because most teams start at the top and skip the bottom.

Component checks. Deterministic assertions on individual pieces. Does the output parse as JSON. Does the extracted invoice number match the checksum. Does the SQL the agent wrote run against a read-only snapshot. Does retrieval return the known-relevant document for a query whose answer you already know. These are cheap, fast, and ruthlessly literal — and they catch most regressions, because most regressions are not subtle. A schema broke. A field moved. A confidence threshold now lets junk through. This is the unit-test layer, and it should run on every change.

Trajectory and end-to-end evals. Full-system runs on a fixed set of cases, graded by a mix of programmatic verifiers and model judges. This is where task success, groundedness, and policy adherence get measured together. It is slower and more expensive, so it runs on a schedule and on every change that touches the prompt, the model, the retrieval configuration, or the tools. The critical property is that it grades the trajectory, not only the final string: which tools were called, with which arguments, in what order. A system can produce a correct answer by an indefensible route, and the route is the thing that will fail next time.

Online measurement. What real users do with real outputs on the real distribution. This is the only layer that cannot be gamed by your own eval set, because the users never read it. It is also the slowest and noisiest, which is why it does not replace the layers below it.

The three layers answer different questions. Component checks ask whether a piece is still doing what it did yesterday. Offline evals ask whether the whole thing satisfies the contract on cases you care about. Online measurement asks whether it satisfies the contract on the cases you did not think of. You need all three, and the most honest signal in the system is the disagreement between the last two — which gets its own section below.

What to measure at each component

End-to-end scores are seductive and nearly useless on their own, because when they drop they cannot tell you where. Instrument the components separately, then use the end-to-end score as confirmation.

Retrieval. Measure it as a first-class subsystem, detached from generation. The standard metrics are recall@k (did the relevant document appear in the top k), precision@k (how much of the top k was relevant), mean reciprocal rank (how high the first relevant hit sat), and nDCG when relevance is graded rather than binary. The reason to isolate it is diagnostic: when the final answer is wrong, retrieval metrics tell you whether the retriever never surfaced the right document or surfaced it and the generator ignored it. Those two failures have completely different fixes, and an end-to-end score conflates them. For when retrieval is even the right architecture in the first place, the companion piece on choosing between RAG and simple search covers that decision.

The answer, given the context. For retrieval-augmented systems, the Ragas family of metrics has become the common vocabulary, and it is worth using even if you do not use the library. Faithfulness asks whether every claim in the answer is supported by the retrieved context — this is the groundedness metric. Answer relevance asks whether the answer actually addresses the question. Context precision asks whether the retrieved chunks that mattered were ranked near the top. Context recall asks whether the information the answer needed was present in the retrieved set at all. Noise sensitivity asks how much the answer degrades when irrelevant chunks are present. Each isolates a distinct failure: an answer can be faithful but irrelevant, relevant but ungrounded, or correct on a clean context and useless on a noisy one.

Generation correctness. Where correctness is checkable, check it — do not judge it. Code is the easy case: run the tests, and report pass@k for the capability view and pass^k for the reliability view. Structured extraction is the underrated case. Schema validity is necessary and nowhere near sufficient: a perfectly valid JSON object carrying the wrong invoice number is a valid failure, and a schema check will pass it every time. Pair every structural check with a semantic one — a cross-field reconciliation, a checksum from the source system, a comparison against a second extraction path.

Classifiers. Treat a classifier as a classifier, not as an LLM feature with a nicer name. The headline metrics are precision, recall, and F1 at the deployed threshold, plus the confusion matrix that shows which errors you are actually making. Under class imbalance, report precision-recall AUC rather than ROC AUC — ROC looks flattering when the negative class dominates. Keep ranking quality and calibration separate: a model can order cases correctly and still be badly miscalibrated, so measure expected calibration error or the Brier score alongside discrimination, because the threshold you choose assumes the scores mean something. Then watch precision and recall drift in production — the threshold is a product decision, and it needs re-examining whenever the class mix moves.

Agents. Grade the trajectory. Task success is the headline, but burying the process hides exactly the failures you need to catch: a tool called with the right parameters by luck, a plan that happened to work, a policy violated in a way that did not change the outcome. The useful component metrics are tool-selection accuracy, argument correctness, and policy adherence graded independently of the final answer — τ-bench treats policy compliance as a first-class criterion, so an agent that books the correct flight while violating the change-fee policy fails, and that is the right call. Then add pass^k on top, because agents are the subsystem where reliability, not capability, is the binding constraint.

One caveat before you over-instrument: every metric you add is a metric someone will optimize. The section on the failure modes of measurement exists for a reason.

LLM-as-a-judge, and its biases

For open-ended output there is often no programmatic verifier, so the industry reaches for a second model to grade the first. It works, it scales, and it is the instrument most likely to be quietly wrong. The canonical MT-Bench study found that a strong judge can agree with human preference over 80 percent of the time — good enough to be useful, not good enough to be trusted unexamined.

The failure modes are well documented and worth naming, because each has a specific mitigation:

• Position bias. The judge favors whichever answer appears first (or second). The effect is strong enough that rankings can be flipped by ordering alone, which means an un-swapped pairwise judge is measuring placement, not quality. Mitigation: grade every pair in both orders and average, or grade pointwise against a rubric.
• Verbosity bias. Longer answers win, whether or not they are better. Mitigation: rubrics that score content rather than length, plus a length-normalized sanity check on the scores.
• Self-preference bias. A judge favors outputs that resemble its own — partly traceable to lower perplexity on familiar text. Judge with the same model that generated the answer and you have asked it to mark its own homework. Mitigation: judge with a different model family than the one under test.
• Limited reasoning. Judges are weak at arithmetic, hard logical entailment, and long chains. Mitigation: never use a judge where a deterministic check exists, and where none does, grade against a reference. Hand the judge the gold answer or a worked solution and ask it to compare, rather than soliciting an unanchored opinion — verification against a known target is a far easier task than open-ended judgment, and pointing the judge at the target routinely rescues a check you would otherwise have had to drop. The judge stays for taste and open-ended quality, never for math.

Two disciplines separate a judge from an opinion with a JSON schema. First, calibrate against humans: hand-label a few hundred examples, run the judge on the same set, and measure agreement. If agreement is low, the judge is not ready, and a high judge score means nothing. Second — and this is the one teams forget — the judge is a model too, subject to the same silent provider updates as the system under test. Pin the judge version and re-check agreement on a schedule, because a judge that drifts will move your quality scores without moving your product.

Human evaluation remains the reference, with one honest caveat: humans are noisy. Before trusting a human-labeled set, measure inter-annotator agreement — Cohen's kappa for two raters, Krippendorff's alpha for more. If two competent people cannot agree on whether an output is good, the task definition is underspecified, and no model will fix that. Low agreement is a signal about the rubric, not about the system.

Non-determinism and statistical honesty

Here is the property that separates AI evaluation from ordinary software testing: the same input does not reliably produce the same output. Even with byte-identical prompts, most model APIs are not deterministic — provider-side batching, mixture-of-experts routing, and infrastructure changes introduce run-to-run variance you do not control. A single run is a sample, not a measurement.

The practical consequence is that each case must be run more than once and reported as a distribution, not a point. Three things follow.

Report intervals, not just averages. An accuracy of 84 percent on 200 samples and 84 percent on 2,000 samples are different claims. For a pass rate, a Wilson score interval is the right tool — it behaves well near 0 and 1, where naive normal intervals fall apart. An eval that reports a bare number cannot distinguish a real change from noise, and it will eventually cause someone to ship a regression they believed was an improvement.

Size the set to the effect you care about. Detecting a small difference needs a large sample; a twenty-case gold set cannot tell 80 percent from 88 percent, however carefully you grade it. This is ordinary experiment design, and it is exactly the point of Evan Miller's Adding Error Bars to Evals: evaluations are experiments, and they deserve the statistics that the rest of science applies to experiments.

Characterize the output distribution, then watch it shift. The regression signal is not "the same input produced a different output" — that happens constantly and means nothing. The signal is a change in the distribution of outputs across repeated replays of the same frozen inputs. Because you own the inputs, you can replay them many times, build a baseline distribution, and treat a statistically significant shift as the regression. That is a hypothesis test, not a diff, and it is the reason you must log enough to replay.

What the ordinary SRE playbook does not account for is the shape of the failure, not the alerting technique. Burn-rate alerting — multi-window, multi-burn-rate — is standard practice and remains the right tool here. What changes for AI features is how the errors arrive. A silent provider update is not independent noise smeared across requests; it hits every request in the affected bucket at once. The failure lands as a step change rather than a slope, and the burn is compressed into a single window instead of accumulating gradually. That is the reason alerting on the rate of burn matters more here than alerting on the running total.

Offline versus online

Your offline eval set is a hypothesis about production, and it is wrong in a specific and predictable way: it is the distribution you imagined, frozen, and then optimized against. Three things happen to a healthy eval set over time. It saturates (the easy cases stop being informative). It overfits (the more you tune prompts against it, the more it measures your ability to tune prompts against it). And it drifts away from the real distribution (production traffic changes; the frozen set does not).

That is what the online layer is for. Explicit signals are the easy half: thumbs up and down, ratings, "report this output," support contacts. Implicit signals are the richer half, and because they are proxies they need naming and skepticism — acceptance and edit distance for generated text or code, regenerate and re-ask rates, escalation-to-human rate, session abandonment, copy-to-clipboard, downstream conversion. Each stands in for "was this useful," and each can lie. Accept-with-heavy-editing and edit-distance-near-zero are opposite readings of the same underlying number, and a suggestion can be accepted simply because fixing it was more effort than rejecting it. The rule is to pair every proxy with a sampled direct judgment, so the proxy stays tethered to reality.

Online comparisons want an experiment: A/B on real traffic where the volume supports it, interleaving when it does not, with the usual guardrails against novelty effects and seasonality. And you need the ability to reconcile the two layers, because their disagreement is the most informative signal you have:

• Online moved, offline did not — the eval set is missing the cases that actually matter. Add them.
• Offline moved, online did not — the metric is not measuring what you think, or you are measuring a bucket users do not hit. Fix the metric.
• Both moved the same way — now you have a finding you can trust.

None of this works without the raw material. Every production request should be logged with its inputs, the retrieved context, the exact model and version, the parameters, the tool calls and their results, and the output. Without that, "replay and characterize the distribution" is impossible, and you are left staring at an aggregate number and guessing. This is also the discipline that makes the guardrails in the agent permissions piece auditable rather than aspirational.

Measuring in production

Offline evals are a gate; production is where you find out whether the gate was set correctly. The instruments that matter there:

Version everything. Model id, prompt hash, retrieval index version, tool manifest version, and the eval baseline — recorded on every trace. Without it, a regression has no suspect, and the incident review becomes archaeology.

Canary before you flip the switch. Run the new configuration on a slice of traffic, compare per bucket against the incumbent, and promote only if the deltas stay within tolerance. Shadow-run it on mirrored traffic when the change is risky enough to justify the cost.

Slice the dashboards. Aggregates hide exactly the cohorts that break: a language, a tenant, a query length, an intent. The average can be flat while one bucket has collapsed, and the collapsed bucket is where the churn is. Report per bucket, and alert on the change in a metric, not its level. Report each quality dimension the same way, because they fail independently: a model regression can collapse task success while schema validity sits at 100 percent, and a schema change can collapse validity while task success is untouched. A single headline number will read as healthy through both. Slice-hunting has a cost the rest of this section is disciplined about: stare at enough slices and one will look broken by chance alone. Decide in advance which slices you alert on, insist on a minimum sample per slice, and correct for the number of comparisons — or treat a suspicious slice as a lead to confirm on a held-out period, not as a finding.

Watch retrieval for drift, separately. Index freshness, new documents that were never re-embedded, and a shifting query distribution all degrade quality through the retrieval path before they touch the model. Retrieval health is a leading indicator, so sample production retrievals and grade them on a schedule.

Treat cost and latency as quality dimensions. Cost per successful task and p95 latency are part of the product, not a footnote. A change that lifts task success by two points while tripling cost per success is a worse system for many products, and a measurement system that does not surface that trade-off is incomplete.

Assume the failure is silent. The dangerous regression is the one where nothing errors, the latency is flat, and the answers are plausibly wrong. No log line catches it. The only instrument that does is periodic sampled human review of real production outputs — a small, boring, ongoing cost that keeps the machine honest and keeps your offline metrics from drifting free of reality.

A minimal eval harness

The whole framework collapses into a small amount of code. What follows is a self-contained harness whose report is the vector the first section argued for: one entry per dimension, each with its own pass rate, its own confidence interval, and its own pass^k. It runs each case several times, keeps every dimension's verdict separate so a strong dimension cannot mask a weak one, treats a dimension a case does not exercise as not applicable rather than failed, reports cost per successful task rather than per call, and gates each dimension on both an absolute floor and a non-inferiority test. It is deliberately small; the point is the shape.

There are two kinds of gate, and the config has to name which is which. An absolute floor is a threshold a dimension must clear no matter what the baseline was — the schema_validity and policy_adherence case, where breaking is breaking even if it was already broken. A relative gate is non-inferiority against the previous run: allowed to move, not allowed to regress past a tolerance. Three details make them honest. Both run per dimension, so a regression in one cannot be netted against a gain in another. The relative gate tests the difference between the two runs rather than comparing one run's interval to the other's point estimate — the baseline carries its own sampling error, and a non-inferiority test that ignores it will pass changes it should reject. And a dimension may legitimately have a floor with no relative gate at all, which is why the two live in separate config keys.

Checking the lower bound has one consequence worth naming: certifying a near-certain floor takes volume. A perfect run's lower bound sits below its point estimate, so a 0.999 floor is not certifiable on a couple of hundred cases — it takes several thousand. When the gate fails because the interval cannot reach the floor, that is a true statement about your evidence rather than a false alarm. Either grow the set, or set the floor where the sample can actually support it. That is why the illustrative floor below is 0.99 and not 0.999: at roughly five hundred samples, a perfect run's lower bound tops out near 0.992, so a 0.999 gate would be one nothing could clear. The stricter number belongs on the continuous production window, where the sample is orders of magnitude larger — a floor should be set where the evidence that will check it actually lives.

The declarations that drive it belong in version control next to the code, so a change to the rubric or the threshold is a reviewable diff.

And in CI, the two commands that matter: run the suite, then gate on the report. The gate reads every dimension's floor — and its optional max_drop — from the same config, so all five thresholds are enforced, not just one.

How much evaluation is enough

Measurement is a cost, and like any cost it needs a budget. Spend it where the expected loss is highest: probability of failure × cost of failure × volume. A rare failure on a low-stakes internal tool can be ignored; a rare failure on something that writes to a system of record cannot. Rank your failure modes by that product, and build the cheapest instrument that covers the top of the list first.

That ordering usually means deterministic checks before judges, component metrics before end-to-end scores, and a small stratified set of hard cases before a large random one. A hundred cases chosen to stress the boundaries — rare intents, adversarial inputs, known past failures — carry more signal than ten thousand sampled uniformly. Then budget the judge itself: grading calls cost tokens, that budget belongs in the config, and it should be alertable like any other spend.

And decide, per metric, whether it gates or merely monitors. Gate on the things that must never break: safety, schema validity, policy adherence, the deterministic floor. Those are absolute floors — they hold regardless of what the baseline was, which is exactly why they are separate from the relative regression gate. Monitor the things that are noisy and expensive: overall satisfaction, taste-sensitive quality. A gate that fires on noise gets disabled, and a disabled gate protects nothing.

Failure modes of measurement

A measurement system has its own failure modes, and they are as expensive as the ones it was built to catch.

• Goodhart's law. The moment a metric becomes a target, it stops being a good metric. If the team's bonus is tied to the eval score, the eval score will improve and the product will not.
• Eval-set overfitting. Tuning prompts against a frozen set turns it into training data. You then cannot measure the thing you just optimized, because your eval no longer predicts anything outside itself. Keep a held-out slice that nobody tunes against.
• Contamination. Public benchmarks leak into training data, so a score on them may reflect memorization rather than ability. Prefer private, freshly authored, or rotating sets for anything load-bearing.
• An uncalibrated judge. A judge with no measured agreement against humans is an opinion with a JSON schema. Its score is a number, not a measurement.
• Proxy divergence. The cheap proxy drifts away from the thing it stood in for. Re-anchor proxies to direct judgments on a schedule, or they quietly stop meaning anything.
• Single-metric collapse. Optimizing one dimension at the expense of the others — task success up, cost up, robustness down. The vector, again.
• Noise mistaken for signal. Acting on a change that sits inside the confidence interval. Without intervals, this happens constantly and looks like progress.
• Multiple comparisons. Eyeballing many slices guarantees a false alarm eventually; with no correction and no pre-registration, slice-hunting manufactures regressions out of noise. Pre-commit the slices you alert on, or confirm a suspicious one on fresh data before acting.
• Telemetry gaps. You cannot replay or diagnose what you did not log. A missing field in the trace is a question you will never be able to answer.
• Composition error. Each step at 95 percent sounds fine until you chain ten of them: 0.95^10 is about 60 percent, so the system fails on two out of every five paths. Measure the composed system, not just the steps.
• Measuring instead of fixing. Dashboards are not verifiers. A team that instruments beautifully but never raises the floor has bought visibility into its own failure.

When measurement doesn't fit

Rigorous evaluation is not always the right investment, and pretending otherwise wastes effort that belongs in the product. The framework above pays off when the task is defined, the volume is real, and the failures are costly. When those do not hold, the honest move is to measure less, not to build a mock experiment.

Before product-market fit, or while a task is still being invented, there is no stable definition of "good," and a formal harness measures a rubric that will be discarded next week. Fast human judgment is the correct instrument there. For genuinely novel open-ended tasks with no ground truth and no defensible rubric, judge-based measurement becomes theater: the number is precise and meaningless. Keep a human in the loop, and measure the outcome humans actually produce — did they accept the output, how much did they change it — rather than a synthetic score for the output itself.

Low-volume, long-horizon tasks — a dozen high-stakes decisions a year — are a poor fit for statistics; there is no sample to power a test. Use structured case review and incident postmortems instead, and resist the urge to compute a rate from eleven observations. And when the cost of measuring exceeds the cost of a human simply doing the task, measurement has lost on its own terms.

One floor holds even here, and it is cheap: log the inputs, the outputs, the model and prompt versions, and the tool calls, and put a deterministic check on anything that writes state. That costs almost nothing, and it is what makes real measurement possible later, once the task has settled and the volume has arrived. Skipping the logging is the one shortcut that is expensive to undo.

Summary

Measuring the quality of an AI system is not the act of producing a score. It is the act of building a small measurement system with the same care you would give any other production subsystem, and it rests on three ideas.

Quality is a vector, not a number: task success, groundedness, policy adherence, safety, robustness, cost, latency, and reliability fail independently, and a single composite hides which one moved. Reliability is not capability: what a system does on its best attempt is a demo, and what it does on every attempt is the product. And model outputs are not deterministic: a single run is a sample, so report intervals, size your sets to the effect you care about, and treat a regression as a shift in a distribution rather than a change in a value.

Around those ideas sit the instruments — cheap deterministic checks first, offline evals with a calibrated judge next, real user signals last — and the discipline that keeps them honest, because the gap between the offline number and the online outcome is where the truth lives. Build the smallest version of this that covers your most expensive failure mode, keep a held-out set that nobody tunes against, and keep one human in the loop to stop the whole thing from drifting. The number you end up showing will be a vector with error bars, and unlike the confident scalar you started with, it will actually be true.