Production AI evaluation: which failures should block a release?

Decide what a better score is allowed to trade away

Consider a support assistant that answers administrators' questions from approved product documentation. A candidate model writes clearer answers than the current model, but sometimes applies instructions from the wrong product version. A higher average quality score could hide that regression if ordinary questions dominate the evaluation.

The release decision needs separate criteria for answer quality and required behavior. In this design exercise, an unsupported product claim blocks approval even when other answers improve. The example contains no model benchmark or production result. The product owner must set the actual requirements and acceptable residual risk for the application.

Write cases around the answer the system owes the user

For a question whose product version is known and whose approved documentation contains the answer, expect a supported answer with a relevant citation. The test should check the claim against the cited material. A citation that exists but does not support the claim is a failure.

For the same question with an ambiguous product version, expect a clarification or the documented handoff behavior. If retrieval returns only a different version's instructions, the assistant must not present those instructions as applicable. Keep this case visible as its own release requirement.

Distinguish an empty successful retrieval from a retrieval failure. Both may prevent an answer, but a timeout should remain an operational failure in the record. Test that the assistant follows the application's recovery or handoff policy and avoids answering from unsupported memory.

Define these outcomes before reviewing the candidate's responses. NIST's AI Risk Management Framework calls for documented test sets and metrics, evaluation under conditions similar to deployment, and explicit limits on how results generalize. The suite must represent the application being released.

Check the grader before trusting the result

Use deterministic checks for properties they can establish, such as whether a required citation identifier exists in the supplied source set. Determining whether an explanation is supported may require a rubric, a judge model, and human review. Keep each check's responsibility explicit.

A judge can accept an unsupported answer or reject a valid paraphrase. Calibrate it against independently reviewed examples, including wrong-version answers and plausible claims that the source never makes. Google Cloud's judge-evaluation guidance compares model ratings with human ratings and exposes disagreements through classification metrics.

Inspect the dangerous false passes directly. Overall agreement can look good when most calibration examples are easy. A disagreement about whether an answer makes an unsupported promise needs resolution against the approved source and rubric; changing judges until one returns a pass would discard that evidence.

Compare the complete application under controlled conditions

Record the model identifier, instructions, retrieval settings, document snapshot, and grader version for each run. Test the path that will serve users. An evaluation that inserts the correct excerpt by hand cannot establish that the deployed retrieval path will find it.

Use the same case set for the current and candidate systems, retain every planned trial, and reset test state between trials. Anthropic's evaluation guidance describes repeated trials for variable model behavior and clean environments to avoid interference between runs. Record provider or harness failures separately so they cannot disappear from a successful-looking average.

Report required-behavior failures by case family, alongside quality judgments and operational results. Include whether the assistant answered, asked for clarification, or handed off. An assistant that avoids every unsupported claim by refusing every question would fail the useful-answer requirement.

Preserve evaluation cases that were not used to tune the candidate. When production incidents reveal a missing case, add it to the next version of the suite and rerun the comparison. Keep prior results attached to the dataset and configuration that produced them.

Make blocking failures an explicit product policy

For this example, approval requires no observed unsupported product claims in the designated release suite, correct handling of version ambiguity, and acceptable answer quality on supported questions. The product owner also defines the latency, availability, and cost limits. Agree on those criteria before the release comparison so a favorable average cannot waive a failed requirement.

Zero observed failures in a finite test suite does not establish a zero failure rate in production. Preserve the size and coverage of each case family, repeat count, and any unresolved grading uncertainty. A narrowly sampled requirement remains narrowly evidenced even if every recorded trial passes.

If a required check fails, repair the system or revisit the intended product behavior through an explicit decision. A changed requirement creates a new evaluation contract; it should not silently relabel a failed run as acceptable.

Carry the evidence into a bounded release

An approval should identify the tested application configuration and its evidence. A later change to the retrieval corpus, prompt, or model can alter behavior and needs review proportionate to that change. Retaining only the model name leaves the release difficult to reproduce.

Use a bounded rollout with an owner who can stop it when required behavior fails. Monitor unresolved handoffs and retrieval failures as well as answer quality, and bring verified failures back into evaluation. NIST includes production monitoring in its measurement framework. The operating plan should specify which signals trigger intervention and how users receive a usable fallback when automated answers stop.