Proof of Quality

The support-desk drafter is a GenAI tool at a European online retailer. It reads an inbound customer-service email, retrieves the relevant policy passages, and drafts a reply for a human agent to review, edit, and send. Before launch, the product team demoed eight drafts to the leadership review. All eight were excellent: correct policy citations, warm tone, no invented promises. The feature shipped. Six weeks later, the support organization's own quality sampling showed that roughly two in five drafts on refund disputes were being rewritten from scratch by agents, and one had promised a customer a refund the policy did not permit.

Nothing went wrong between the demo and production. That is the uncomfortable part. The demo was an accurate sample of the system — an accurate sample of its upper tail. A generative model does not have a quality; it has a quality distribution, and any particular output is a draw from it. When you look at eight outputs you have chosen to show other people, you are not measuring the distribution, you are reporting its maximum. The distinction is the whole of this guide compressed into one sentence: a demo answers “can it?” and shipping requires “how often, and how badly when not?”

Two things follow immediately. First, the sample size that matters is not how many outputs you looked at but how many distinct inputs you looked at, drawn how. Eight outputs for the same easy intent is a sample of one input class. Second, the interesting statistic is almost never the mean. Agents do not experience your average draft; they experience the specific draft in front of them, and the reason they stop trusting the tool is the tail. A feature whose mean rose and whose fifth percentile fell has usually gotten worse.

Traditional software degrades in ways your instincts already cover: someone changed the code, the change was reviewed, a test failed or did not, and if quality dropped there is a commit to point at. LLM systems break along three additional paths, none of which produce a diff you would recognize.

Output variance is the same input yielding different outputs across calls. Even at temperature 0 you do not get determinism — batching, hardware, and provider-side routing make repeated calls merely similar. The practical consequence is that a single passing run of a single case proves nothing about that case, which is why eval design has to think in sample sizes even when the input set is fixed.

Distribution shift is the gap between the inputs you tested and the inputs you serve. The drafter's eval set was written by the product team in English, one issue per email, politely phrased. Actual inbound mail arrives in Danish and German, mixes a delivery complaint with a refund demand and a GDPR erasure request in one message, and is frequently furious. The system did not change. The input distribution did — and it keeps moving as marketing runs a campaign, as a courier fails in one region, as a policy change generates a new complaint genre.

Silent regression is a quality drop with no failing test and no code change. Someone tightened the system prompt to fix one complaint and traded away citation discipline. The retrieval index was rebuilt with a new chunking strategy. The provider moved the alias you pinned to. Each of these changes system behavior; none of them passes through the review path your organization built for code.

Here is the reframe the rest of this guide depends on. In a deterministic system, the specification and the tests are separate artifacts: the spec says what should happen, the tests check a sample of it, and a careful reader can verify the code against the spec directly. In a generative system there is nothing to read. The prompt is not a specification of behavior; it is an attempt to induce behavior, and whether it succeeded is an empirical question. The only place a quality requirement can be stated in a form that survives contact with reality is an eval.

Which means: an unmeasured quality bar is an unstated requirement. “Drafts should be good enough to send” is not a requirement; it is a mood shared by people who have not yet discovered that they disagree. Watch what happens when you force it into eval form. Good enough to send becomes four separable claims — the draft states nothing factually false about the customer's order; the draft cites the applicable policy section and promises nothing outside it; the draft discloses no personal data beyond what the customer already supplied; the draft's register matches the brand's tone guide. Each claim now has a possible measurement, a possible threshold, and a possible disagreement that can be resolved with data rather than seniority.

This reframe also fixes an organizational pathology you have probably already met. When quality is a mood, quality debates are status contests — whoever is most senior or most insistent in the review decides what ships. When quality is a set of dimensions with thresholds, the debate becomes a product trade-off with a cost attached: raising the policy-adherence bar from 0.90 to 0.96 costs you a slower model, tighter retrieval, or a narrower launch. That is an argument a team can actually have. Module 7 develops this into a working practice.

One more idea before the practice starts, and it is the one that keeps the rest honest. An eval does not verify that a system works. An eval is an attempted refutation: a specific attempt to make the system fail in a specific way. When it passes, the only thing established is that this particular attempt did not succeed. When your whole suite passes, the honest statement is “under these attempted refutations, on this dataset version, the system did not fail” — not “the system is correct.”

This is not philosophical decoration; it changes three decisions you will make this quarter. How you value a suite: its worth is measured by what it could catch, not by the score it produces. A suite of 400 easy cases scoring 98% is worth less than a suite of 120 vicious ones scoring 79%, because only the second is trying to fail. What you do after an incident: every production failure is a refutation you did not think to attempt, so it becomes a permanent case in the suite before the fix merges — the regression ratchet of module 6. How you phrase claims: “no detected policy violations in 400 stratified cases including 60 adversarial PII probes, dataset v4, rubric v2.1” is a defensible sentence. “Our evals pass, so it is compliant” is a sentence that will be read back to you.

Notice what this framing does not license. “Evals can never prove correctness” is not an argument for skipping them, any more than “tests never prove program correctness” is an argument against tests. It is an argument for building suites that try hard to fail, reporting results with their scope attached, and treating the suite as a permanently unfinished artifact that grows every time reality finds something you missed.

Every quality check you will ever write sits on one of four levels, ordered by how much human judgment it approximates and how much it costs to run.

Assertions are deterministic pass/fail checks on the output itself: does it parse as JSON, is it under 200 words, does it contain the required disclosure sentence, does it avoid a banned phrase. Programmatic checks compute a measure without model judgment: regex or NER detection of personal identifiers, verifying that every cited policy section exists in the corpus, computing recall@k for a retrieval stage. Model graders — LLM-as-judge — apply a rubric with a model doing the reading, which is the only cheap way to grade things like faithfulness and tone. Human panels are multiple trained people grading against written guidelines; they are the fidelity ceiling and, crucially, the thing every judge's trustworthiness is borrowed from.

The single most useful consequence of this picture: level 4 is not optional even though you can barely afford it. Human panels are what levels 3's numbers mean anything against. Drop them and your judge scores become an internally consistent fiction — reproducible, precise, and unmoored from whether anyone would actually accept the draft.

Levels 1 and 2 catch the failures that require no judgment at all, and most teams build a tenth of what they should. They are free to run, so they run on every case in the suite and on every output in production; they never drift; they never need calibration; and when they fail they tell you exactly what is wrong. Here is the drafter's actual deterministic layer.

Read the fourth column across, because it is the argument for the levels above. Every deterministic check is exact about a narrow thing and blind to the thing next to it. The PII check catches an IBAN that appears from nowhere; it cannot catch a sentence that restates a customer's health situation in plain words, because no identifier appears. That single row is the boundary between level 2 and level 3, and it is where most of the real risk lives.

Level 3 exists because the most important dimensions of quality are matters of judgment. Is the draft faithful to the customer's actual issue, or has it answered a nearby question? Does the tone match a brand guide that says “warm, direct, never defensive”? Does it commit to something the cited policy does not authorize? None of these reduces to a match or a computation, and all of them are the reason support agents rewrite drafts.

A model grader scales human-like judgment at roughly one part in two hundred of human cost. That is the whole value proposition, and it comes with a condition attached: a judge is only as trustworthy as its demonstrated agreement with humans. An uncalibrated judge is not a cheap human; it is a confident stranger with a rubric. Module 4 is entirely about turning the second thing into the first, so for now, take the dependency as structural: level 3's fidelity is not a property of level 3.

Level 4 is where the standard lives. It is also where the arithmetic becomes brutal, so do the arithmetic explicitly. Grading 1,000 drafter outputs on four dimensions:

The bottom row is the one that reorganizes your thinking. You will never human-grade a thousand examples, and you do not need to. You need humans on a calibration sample — 100 to 150 items, chosen to span the rubric's range — so that the 40-euro judge run can inherit their authority. Human effort is spent on establishing the measuring instrument, not on doing the measuring.

Here is the method, run end to end on the drafter. Start from the four rubric dimensions and refuse to stay at that altitude — enumerate the concrete failure classes underneath each, then assign each class the cheapest level that reliably detects it.

Two things fall out of the table. First, dimensions split across levels — PII handling is not “a judge dimension”; it is one cheap check and one expensive one, and running only the cheap half is how teams end up confidently wrong. Second, the judge's workload is now four narrowly scoped questions instead of one holistic “is this good,” which is both cheaper and more reliable to calibrate.

The hierarchy picks your grader. A second, independent choice picks your target: are you scoring one stage of the pipeline in isolation, or the whole thing as the user experiences it?

A component eval scores a single stage against a stage-local ground truth. The drafter's retrieval stage is the natural example, and it is our secondary running case: over the compliance corpus of Guide Nº 02, a labeled set pairs 180 customer questions with the policy chunks a domain expert marked relevant, and the metric is recall@k — the fraction of truly relevant chunks appearing in the top k retrieved. Recall@5 of 0.86 means that 14% of the relevant policy chunks never appeared in the model's context at all. No prompt engineering fixes that; the information was not there.

An end-to-end eval scores the final draft. It is the only thing that validates the actual experience — retrieval can be perfect while the generation ignores what it retrieved — but when the number moves it cannot tell you which stage moved it.

A golden dataset is a versioned, stratified collection of inputs, each paired with what must be true of a good output. The phrase people expect — “input plus correct answer” — is wrong for generative systems in a way worth being precise about. For a classifier, the label is the correct output and exact match is meaningful. For a drafted customer email, there are thousands of acceptable replies and no canonical one; scoring against a single reference string measures similarity to one author's phrasing, not quality.

So the drafter's golden set stores properties. A case looks like this:

case_id: gs-0187
lang: de
intent: refund-dispute
difficulty: hard
source: production-sanitized (2026-04-11)
input:
  customer_email: "Ich habe die Jacke am 3. April zurückgeschickt ..."
  order_record: { order_id: "NL-88431", status: "return_received",
                  category: "outerwear", purchased: "2026-03-02" }
expect:
  must_cite: ["returns-policy-v12#4.2"]
  must_not: ["promise refund before inspection",
             "state a refund date"]
  must_state: ["inspection window is 5 working days"]
  dimensions: { policy_adherence: ">=3", pii_handling: "==4" }

This is reference-free grading: the case constrains the output space without prescribing a point in it. Note that the constraints are heterogeneous by design — must_cite is checked by a level-2 programmatic check, must_not entries are judged, and the dimension floors come from the rubric in module 4. The dataset and the hierarchy are the same design viewed from two directions.

Four channels, four distinct blind spots. A set drawn from one channel inherits that channel's blind spot completely, which is why a serious set uses all four in known proportions.

Sanitized production traffic is the only source that is representative by construction, and it is the one with a legal gate in front of it. Real customer emails are personal data; using them as test data is a new purpose, which needs a lawful basis, a data-minimization pass, and usually pseudonymization before the case ever lands in a repository engineers can read. Practically: strip or replace names, addresses, order numbers, and any special-category content; keep the structure and the fury, because those are what you are trying to capture. Skipping this step is not a corner cut, it is a reportable one.

Synthetic generation is fast, unlimited, and quietly self-similar. Ask a model for 200 angry customer emails and you get 200 variations of one angry customer: grammatical, single-issue, and suspiciously articulate. It is genuinely useful for filling combinatorial gaps (“generate this scenario in Danish with a typo rate matching production”) and for adversarial probes, but as the primary source of record it produces a set on which everything passes.

Expert-authored hard cases are the ones nobody else writes: the customer who quotes a policy section incorrectly and demands you honor their reading; the message that mixes a refund demand with a GDPR erasure request; the case where the retrieved context contains a different customer's ticket. This is where domain experts — senior support agents, your privacy counsel — earn their eval budget, because these cases require knowing what goes wrong, not just what happens.

Incident-derived cases are the highest-value cases you will ever have, because reality wrote them. Every production failure becomes a permanent case before the fix merges. Module 6 develops this as the regression ratchet; the point here is that it is a sourcing strategy, and over two years it becomes the most valuable third of the set.

An aggregate score is a weighted average over your dataset's composition, which means you chose the answer when you chose the cases. If 70% of your set is order-status lookups because those were easy to collect, your headline number is mostly a measurement of order-status lookups, and it will remain reassuring while the hard intents rot.

Stratification is the fix: define the slice dimensions that matter, set a target count per cell, and report per-slice as the primary view with the aggregate as a footnote. For the drafter the dimensions are intent (8 values), difficulty (routine / hard / adversarial), language (en / de / da), and risk category (whether the case touches payment, health, or a third party). You do not need every cell of the cross-product; you need every cell that carries risk, and enough cases per cell that a slice score is not noise — 30 to 60 is the working range, below which slice movements are unreadable.

Deliberately over-weight the hard parts. A set that mirrors production exactly gives you 2% adversarial cases, which at n=400 is eight cases and no statistical power where the risk is concentrated. Over-weight, then re-weight when you want an estimate of the production mean.

Contamination is your test set finding its way into the thing being tested. It moves in two directions, and both produce scores that rise while quality does not.

Outward leakage is the one you control and the one that actually bites. An engineer needs few-shot exemplars, opens the eval set, and copies the three clearest cases into the prompt. The suite's score jumps four points overnight. Nothing improved: the system now has the answers to three of its own exam questions, and — worse — those three cases are now permanently uninformative, because they will pass forever regardless of what else breaks. The same happens more severely when eval cases are swept into a fine-tuning corpus.

Inward contamination is the base model having seen your test data during pretraining. This mostly matters if you build from public benchmarks; a set sourced from your own production traffic and internal policy corpus is largely immune, which is one more reason to build your own. The tell is a suspiciously high score on a slice nobody optimized, sometimes with verbatim familiarity in the output.

A golden set is a snapshot of a moving world. The product ships new intents, the returns policy gets a new section, a courier failure invents a complaint genre, and the set that was representative in January is a museum by September. But refreshing carelessly destroys the only thing that made scores useful — comparability across time.

Refresh on triggers, not on the calendar. The drafter's triggers: a new intent ships; the policy corpus publishes a new version; an incident postmortem closes (its case merges immediately); the quarterly drift review finds a production slice with under-representation greater than a stated tolerance. Calendar refreshes produce churn in quiet quarters and complacency in loud ones.

Then version every change, and treat a version bump as a new baseline, not a movement. This is the discipline people skip, and it produces the most embarrassing conversation in module 8: the team that added 40 easier cases in March and reported April's higher score as product improvement. The rules that prevent it are mechanical — the set is immutable once tagged; changes create v5; any score is reported as score @ set-v4 / rubric-v2.1; when the set changes, re-run the previous system version against the new set to establish the new baseline before claiming any delta.

Finally: the set has an owner, named in the same place the on-call rotation is named. Unowned datasets rot in a specific, predictable way — nobody merges the incident cases, nobody runs the drift review, and eighteen months later the suite is measuring a product that no longer exists.

An LLM-as-judge setup is a model reading an output and applying a rubric to it. It works — well enough to carry most of the grading in a serious eval program — for a specific reason: judging is an easier task than generating. Deciding whether a draft promised a refund the cited policy does not authorize is a reading-comprehension question with the evidence in front of you, and models are better at that than at writing a flawless draft in the first place.

But “it works” is a claim about a configuration, not about the technique. The same judge prompt with a different rubric wording, a different model, or a different output format produces measurably different scores. That is what makes it an instrument: readings are a function of the apparatus, and you do not know what the apparatus does until you compare it against a standard.

The cost of skipping this is not that the numbers are wrong in a random direction. It is that they are wrong in a consistent direction — an uncalibrated judge typically runs generous, because rubric-following models grade charitably and because the failures that matter most are the subtlest ones. You get a stable, precise, reproducible 4.6 out of 5 that tracks nothing anyone would recognize as quality, and it is very difficult to argue with a stable number.

A rubric is the measurable form of “good.” Three properties separate one that works from one that produces noise.

Dimensions must be independent and observable. If two dimensions move together on every example, you have one dimension and a redundant grading cost. If a dimension cannot be decided from the artifact plus its context — “is this draft helpful?” depends on what the customer does next — it belongs in online evaluation (module 5), not in a rubric.

Score levels must be anchored to observable properties, not adjectives. This is the single highest-leverage decision in the module. A 1–10 scale with no anchors is astrology with decimals: the grader — human or model — invents its own mapping, that mapping drifts between items, and two graders never share one. Anchors say what is true of the artifact at each level.

Prefer few levels. Four is usually right: fail, partial, pass, exemplary. Ten-point scales invite false precision and collapse in practice to three clusters anyway.

Here is the drafter's rubric in full. It is binding on everything downstream in this guide: the judge prompt in the next section embeds it verbatim, the graded examples apply it, and the thresholds in module 6's gate are stated against it.

A rubric only becomes real when it is applied to hard cases. Here are three drafter outputs graded against it — a pass, a fail, and the borderline case, which is the one that teaches. All three are German refund-dispute cases, sanitized, shown in English for readability.

Two lessons from the trio. First, the fail case had good tone — a single aggregate score would have averaged 1, 1, 1, 3 into something that sounds mediocre rather than dangerous, which is the argument for per-dimension reporting made concrete. Second, the borderline case is where rubrics get built. Grade only clear cases and your rubric will look excellent and decide nothing; the disagreements are the signal telling you which anchor is underspecified.

Here is the drafter's judge prompt, complete. It is one dimension per call; the runner makes four calls per draft and assembles the results. Every element in it is load-bearing, and the reasoning follows the prompt.

SYSTEM
You are a quality grader for customer-service email drafts at a European
online retailer. You do not write or improve drafts. You apply one rubric
dimension to one draft and report a score with quoted evidence.

You grade ONLY the dimension named in the user message. Ignore every other
quality aspect, including ones you consider more serious. If the draft has a
severe problem outside your dimension, that is another grader's job.

Evidence rule: every score you give must quote a verbatim span from the DRAFT
that justifies it. If you cannot quote a span, you may not assign a score
below 4 on that basis. Do not quote from the policy or the order record as
evidence for the draft's behaviour; quote the draft.

Output valid JSON only, matching this schema, with no surrounding prose:
{
  "dimension": string,
  "score": 1 | 2 | 3 | 4,
  "anchor_matched": string,     // the anchor text you applied, verbatim
  "evidence": [string],         // verbatim spans from the DRAFT
  "justification": string       // <= 40 words, referring to the evidence
}

USER
DIMENSION: policy_adherence

ANCHORS (apply exactly; choose the highest level fully satisfied):
4 - Cites the applicable section, states its conditions accurately, and
    explicitly declines any remedy outside it.
3 - Cites the applicable section and commits to nothing beyond what it
    authorises.
2 - Commits only within policy but cites the wrong or no section, so the
    agent cannot verify it; or converts a conditional process into an
    unconditional timeline.
1 - Commits to any remedy, timeline, or exception the cited policy does
    not authorise.

CUSTOMER MESSAGE:
<<<{{customer_message}}>>>

ORDER RECORD:
<<<{{order_record_json}}>>>

APPLICABLE POLICY TEXT:
<<<{{retrieved_policy_sections}}>>>

DRAFT TO GRADE:
<<<{{draft}}>>>

Return the JSON object now.

RUNTIME: temperature 0, top_p 1, max_tokens 400, seed fixed, judge model
pinned to an explicit version string (never an alias).

Why one dimension at a time. Holistic grading lets the model form an overall impression and then rationalize four numbers consistent with it — which is why the fail case above would score 2-2-2-2 under holistic grading instead of 1-1-1-3. Separate calls cost four times as much and are worth it, because the per-dimension signal is the thing you actually act on.

Why quoted evidence is mandatory. Requiring a verbatim span from the draft forces the judge to locate the violation rather than assert it, and it gives you an audit trail: when you spot-check the judge, you are checking whether the quoted span actually supports the score, which a human can do in seconds. It is also the specific mitigation for rubric sycophancy in the next section.

Why the anchor is echoed back. anchor_matched costs nothing and turns a disagreement into a diagnosis: if the judge and a human both say 2 but matched different anchors, they agree by coincidence and the rubric has a defect you would otherwise never see.

Judge-model selection. Three criteria. Use a capable tier — judging is cheaper than generating but not free, and small models fail most on exactly the subtle policy distinctions you care about. Use a different model family from the generator, to avoid self-preference. And pin the version explicitly: the judge is your measuring instrument, and an instrument that silently updates itself invalidates every historical number. At roughly €0.01 per grade, four dimensions across a 400-case set costs about €16 per gated run — cheap enough that the cost argument for holistic grading never survives contact with the arithmetic.

Judges fail in documented, repeatable ways. These are not model quirks that the next generation will fix; several are structural consequences of asking a language model to compare or score text. Each has a mitigation that costs something, and the cost is the reason teams skip it.

Pairwise versus absolute scoring is the design choice that decides which of these you will fight. Absolute scoring — the drafter's approach — grades one artifact against anchors, produces a score you can threshold in CI, and is vulnerable to leniency drift and verbosity bias. Pairwise comparison — “which of these two drafts is better” — is more sensitive to small quality differences and far more natural for the model, which makes it the better tool for A/B-ing two prompt variants; but it produces a relative ordering you cannot gate on, and it is where position bias lives. Most mature programs run both: absolute scores for the gate, pairwise for deciding whether change B beats change A.

Calibration is where the judge's authority comes from, and it runs in a fixed order that most teams get backwards. Humans must agree with each other before you ask whether the judge agrees with humans. If two trained annotators grade the same drafts and reach κ ≈ 0.3, there is no target to hit — the rubric is ambiguous, and a judge that matched those humans would be matching noise.

The pass, step by step, as it runs for the drafter:

1. Draw the sample. 120 cases from the golden set, stratified to span the rubric's range — deliberately including hard and adversarial cells, because a sample of easy cases produces high agreement that tells you nothing about the boundary you care about.

2. Write annotator guidelines. The rubric plus worked examples at every level plus the adjudicated borderline cases. This document is the real deliverable; the annotators are applying it, and its ambiguities become your disagreements.

3. Two annotators, independently. Two senior support agents, blind to each other and to any judge output. Roughly four minutes per item across all four dimensions.

4. Compute inter-rater agreement first. Cohen's κ per dimension. The drafter's gate: κ ≥ 0.7 to proceed. First run produced κ = 0.62 on policy adherence, which sent the team back to the anchors — the timeline clause from the borderline case above came out of exactly this loop — and the rerun reached 0.78.

5. Run the judge on the same sample, same rubric version, blind to the human labels.

6. Compare per dimension, never in aggregate: exact-match rate, within-one-level rate, and directional bias (does the judge run generous or strict, and where). A judge at 0.84 exact agreement overall might be at 0.94 on PII handling and 0.71 on tone, and only the second number tells you what to do.

7. Route the disagreements. Every disagreement is diagnosed into one of three buckets: the rubric is ambiguous (fix the anchor, re-run), the judge prompt is underspecified (fix the prompt, re-run), or the human was wrong (fix the guidelines, retrain the annotator). Averaging the judge and human scores is not on the list.

8. Accept at a documented threshold, with a dated report. The drafter accepts a dimension at exact agreement ≥ 0.80 with no systematic directional bias, and records κ, agreement, sample composition, date, and all three version strings. That report is the artifact module 8 puts in front of an auditor.

The two instruments answer different questions. Offline: did the system fail any of the ways we anticipated, on a fixed population, under a controlled grader? Online: is the system producing the outcomes we want, on the population that actually exists, right now?

Neither substitutes for the other, and the reasons are asymmetric. Offline cannot see distribution shift, novel failure genres, or anything about what humans do with the output — but it is reproducible, it can gate a release before users are exposed, and it can be run against a candidate that has never served traffic. Online sees everything real, but it only sees it after users have experienced it, it cannot isolate causes without an experiment, and its signal is noisy and lagged.

The loop between them is the actual system. Production surfaces a failure genre nobody imagined — customers forwarding a competitor's email into the thread and the drafter dutifully summarizing it — and that becomes three golden cases, a rubric clause, and a permanent offline check. The offline suite is where production's lessons are stored so they are never re-learned. A team with excellent dashboards and no path back into the golden set is re-discovering the same failures indefinitely.

The best online metrics are written by user behaviour, because behaviour is expensive to fake and independent of your grading apparatus. For the drafter, the agent's own actions are a continuous, free, high-volume quality signal:

Note the property they share: each one costs someone something. That is what makes them hard to game and hard to fool yourself with. Compare the two metrics teams reach for first.

One caution on outcome metrics generally: they are lagging, and some of them are lagging badly. Reopen rate needs seven days; complaint rate needs longer and is confounded by everything else the company does. That lag is precisely why guardrails and canaries exist — you cannot run a release on a signal that arrives next week.

A guardrail metric is a dimension that must not regress no matter what else improves. It is not the metric you are trying to move; it is the one whose movement ends the experiment. The drafter's are: PII-detector hit rate, policy-violation rate on the judge's level-1 anchor, escalation rate, and the rate of drafts containing an unauthorized commitment. Each carries a threshold and a pre-agreed response, decided before the change ships, because deciding during an incident is how a bad change survives for nine days on the strength of someone's optimism.

A canary is the release mechanic that makes guardrails useful. Route a small traffic slice to the new prompt or model, watch the guardrails on a short window, and auto-revert on breach. The key design parameters are the slice size (large enough to detect the effect you care about within the window), the window (long enough for the metric to be readable), and the revert being automatic — a guardrail that pages a human who then reasons about whether it is real is a guardrail with a human's optimism inserted into its control loop.

Judge-grading every production output does not survive arithmetic. At 50,000 drafts a day and four dimensions at €0.01 per grade, full coverage is €2,000 a day — more than the feature costs to run. So you sample, and how you sample determines what you can see.

Random sampling gives an unbiased estimate of the population mean. It is the right base layer and it is nearly useless for rare events: a 2% random sample of a slice that is 1% of traffic yields 10 items a day, and a failure rate within that slice is unreadable for weeks.

Stratified oversampling fixes exactly that. Define the risk-bearing slices, sample each at a rate proportional to risk rather than volume, and re-weight when you want a population estimate. Health-data mentions are 1% of drafter traffic and perhaps 40% of its regulatory risk; sampling them at 25% instead of 2% costs almost nothing in absolute terms and is the difference between detecting a problem in a day and detecting it in a quarter.

Targeted sampling grades outputs that something already flagged: a thumbs-down, an agent rewriting a draft from scratch, a level-2 PII check firing, a low generator confidence, an escalation. These are pre-enriched for failure, so they are the cheapest place to find real problems — with the standing caveat that they are a biased sample and cannot be used to estimate a rate.

When you experiment on a generative feature, the experimental design is the same discipline it always was — randomization, sample size, run length, multiple-comparison hygiene. Three things are genuinely different, and those are what this section covers.

Movement is per-dimension, and a single success metric hides trades. Prompt v8 raised the tone dimension by 0.3 and lowered policy adherence by 0.2. Under one blended “quality score” that is a modest win; under per-dimension reporting it is a trade between the thing agents complain about and the thing legal cares about, which is a decision, not a number. Always report the vector.

Variance is high and novelty effects are real. Generative output varies per call, users adapt their behaviour to a changed tool (agents who learn the new drafts need less editing look like a quality gain that is partly a learning curve), and effect sizes are often small relative to that noise. Run longer than feels necessary, and be suspicious of the first two days.

Guardrails are stop conditions, not metrics to trade against. An experiment that improves your primary metric and breaches a guardrail has not produced a trade-off to discuss; it has ended. This is worth writing into the experiment template, because in the moment there is always an argument that the guardrail breach is small and the win is large.

An eval gate runs the suite on a proposed change and decides whether it may proceed. The design problem is not running it; it is choosing thresholds that block real regressions without firing on noise — because a gate that fires on noise gets overridden, and a gate that is routinely overridden is decoration with a latency cost.

Per-dimension semantics. Not every dimension deserves the same power. The drafter's gate: PII handling is a hard block — any case scoring 1 fails the build, no threshold arithmetic involved, because a single third-party disclosure is not averaged away. Policy adherence is a banded block — the slice mean must not fall more than one band below the recorded baseline. Tone is a warn — reported in the PR, blocks nothing. Latency and cost are informational.

Bands, not points. At n = 400 with judge nondeterminism, a two-point move in a percentage is frequently noise. Establish the band empirically rather than by intuition: run the unchanged system against the suite ten times and observe the spread. The drafter's per-slice policy-adherence mean varies by about ±1.5 points run to run, so the gate blocks at a drop greater than 3 points on any reported slice, and — critically — on any single slice, not on the aggregate, since figure 3.1 already showed what the aggregate hides.

Nondeterminism discipline. Sample each case three times and threshold on the mean; a case that passes twice and fails once is a signal, not a flake to be retried away. Flaky gates train people to hit re-run, and a re-run culture converts a control into a delay.

An override protocol, because sometimes the gate is wrong. Overrides are legitimate — a rubric change intentionally lowers scores, a slice regression is a known trade already agreed. What makes them safe is that they are named: an override requires a stated reason, a named approver who is not the change's author, and an entry in the gate log. Module 8 will point out that this log is also the change-management evidence an auditor already knows how to read.

The gate is only as good as its trigger list, and the default trigger list — “when code changes” — misses most of the surface. Everything in this table changes what a customer reads:

The operating rule is one sentence: the prompt is config, and config changes are deploys. Everything follows — prompts live in version control, changes go through review, the gate runs, and the console is read-only in production. The last clause is the one that gets resisted, because editing a prompt in a provider console is the fastest way to respond to a complaint and it feels like customer service rather than deployment.

Provider deprecations make model upgrades mandatory and put them on someone else's calendar. That is the argument for a playbook: you will run this under time pressure, more than once a year, and the quality of the decision should not depend on who happens to own it that quarter.

1. Pin, always. The precondition. If your config points at a floating alias, you are not upgrading — you are being upgraded, at a time you did not choose, with no diff to inspect.

2. Check the instrument before the subject. If the judge model is also changing — because it is from the same provider, or because you were going to upgrade it too — stop and recalibrate the judge first, as a separate change. Moving generator and judge together produces an uninterpretable number, and the direction of the error is unknowable.

3. Shadow-run the candidate against the full golden set, plus a mirrored sample of recent production traffic. No users are served; you are producing a diff.

4. Diff per slice, not aggregate. A new model version is not uniformly better or worse — it is better at some things and worse at others, and the aggregate averages that into a number that hides both. The drafter's actual last upgrade: aggregate policy adherence up 1.1 points, German refund-dispute slice down 6 points. Aggregate said ship.

5. Fix or accept explicitly. The German slice regression was traced to the new model being less willing to state conditions bluntly; a prompt adjustment recovered it. Had it not, the choice — accept the regression, delay, or launch with that slice routed to the old pin — would have been made and written down rather than absorbed.

6. Staged rollout behind guardrails (module 5), then 7. documented sign-off: what ran, against which versions, who approved, what was accepted.

The final piece is a process rule with an outsized payoff: every production failure becomes a permanent golden case before the fix merges. Not after; before — because after means never, reliably, in every organization.

The regression ratchet has three effects that compound. The suite grows monotonically harder, so its value increases rather than decaying. It becomes institutional memory that survives turnover: the engineer who understood why the drafter must never state a refund date leaves, and the case they wrote stays. And it converts incidents from pure cost into an asset, which changes the emotional register of a postmortem — the question “what case does this become?” is answerable and blameless.

Make it mechanical. The postmortem template has a required field for the case ID. The fix PR references it. The gate checks that the referenced case exists and fails on the new case before the fix and passes after — which is, precisely, a regression test written in the language of your eval suite.

You already practise this discipline. A user story with acceptance criteria written before the sprint is the same idea: state what would count as done, in advance, in terms someone other than the author can check. Eval-driven development is that practice adapted to a system whose output cannot be checked by reading it.

The reason order matters is not moral, it is epistemic. Consider two versions of the same artifact. Written first, the eval says “a draft for a customer disputing a refund on a non-returnable item must decline clearly, cite §4.4, and offer the exchange alternative” — a statement about what the product should do, which the implementation may fail. Written after, the same team writes cases sampled from what the system currently produces, notices that declines are usually accompanied by an apology and a policy link, and encodes that. The second eval will pass forever and never once tell them the product is wrong.

The second effect of writing first is that it forces the specification to be finished. “Handle refund disputes” survives a roadmap review indefinitely. Writing the eval first makes you answer, this week, what the system should say when the customer is not entitled to a refund but has been a customer for six years and is furious — which is the question the feature was always actually about, and which will otherwise be answered at 4pm on a Friday by whoever is holding the prompt.

Here is the cycle run end to end on a real increment: adding the German refund-dispute intent to the drafter. Note that this is a new slice, not a new system — most EDD in practice is incremental, and the loop is the same at both scales.

Define the rubric delta. The four dimensions hold; what changes is that refund disputes surface a boundary the anchors did not cover — declining a remedy the customer believes they are owed. Two anchors gain clauses, producing rubric v2.1. Because the rubric moved, the judge is recalibrated against the 120-case sample before anything else proceeds.

Build the golden slice. 60 cases: 30 sanitized production disputes, 18 expert-authored (including the six-year customer, the misquoted-policy customer, and the dispute-plus-erasure request), 12 synthetic variants filling difficulty cells. Slice refund-dispute·de, in golden set v5 — v4's 400 cases plus this slice, 460 in total, which is the additive new-cell bump module 3's versioning table describes. Nothing is implemented yet, and the current system scores 41% on it — figure 3.1's number, which is where this whole thread started.

Implement. Prompt work, retrieval tuning for the returns policy sections, an added post-processing check for refund-date claims.

Grade. Run, read per-dimension results, iterate. This inner loop runs several times a day and is the fast cycle.

Ship gate. The thresholds agreed at definition time: policy adherence ≥ 3 on 95% of the slice, no PII score below 4, tone ≥ 3 on 90%. Note these were agreed before the numbers existed, which is what makes them a decision rather than a rationalization.

Monitor. The slice gets its own dashboard row and its own guardrail; module 5's mechanics apply from day one, not after the first incident.

Refresh. Production surprises — customers quoting a competitor's policy at us, disputes arriving in Danish, a case where the customer's stated reason for return was medical — become new cases, which is where the loop closes and the next rubric delta comes from.

The eval's second job is political, in the useful sense. Quality debates on GenAI features have a characteristic failure mode: two senior people read four outputs each, form opposite impressions, and the disagreement has no resolution procedure because there is nothing in it that could be checked. Whoever cares more, or ranks higher, wins.

A rubric with thresholds converts that into a different kind of argument. “Is it good enough?” becomes “policy adherence is at 0.93 on the refund slice and our threshold is 0.95; do we hold the launch two weeks, ship with the slice routed to human-only drafting, or lower the threshold and say why?” That is a product decision with visible costs, and it is resolvable — not because the number is objective truth, but because everyone is now arguing about the same thing.

Two mechanics make this work in practice. Attach the cost curve. Raising a threshold is never free: it costs latency, spend, coverage, or launch date. The drafter's policy-adherence bar could go from 0.95 to 0.98 by adding a verification pass that reads the draft against the cited policy and regenerates on conflict — €0.004 per draft, 900ms of added latency, and roughly 3% of drafts falling back to human drafting. Presented that way, the threshold becomes a trade a group of adults can make. Set weights explicitly. When dimensions conflict — the tone-versus-policy-adherence trade from module 5 — the resolution is a stated priority, not an averaged score. The drafter's stated order: PII handling is inviolable, policy adherence is next and gated, factual accuracy is gated, tone is optimized within those constraints. That ordering was a product decision, made once, in daylight.

The last use of an eval is the one that saves the most time, and it happens before anything is built: if you cannot construct the eval, you cannot verify the feature, and that is a scoping finding rather than a footnote.

When a proposed requirement resists being written as a rubric dimension, one of three things is true, and they have different remedies. It is underspecified — “delightful” has not yet been decomposed, and the work is to decompose it into observable properties (which usually reveals that the stakeholder meant two specific things). It is not observable from the artifact — “the customer feels heard” depends on what the customer does next, so it belongs in online evaluation as an outcome metric, not in an offline rubric. Or it is genuinely not measurable yet, in which case shipping it means shipping something you cannot tell is working, and the honest move is to de-scope it from v1 and say so.

A related distinction saves a great deal of misdirected effort. A capability question — can the model do this at all? — is answered by a spike: twenty examples in an afternoon, and the answer is usually obvious. A quality question — how reliably, on which inputs, failing how? — is answered by an eval and cannot be answered any other way. Teams routinely run a spike, get a promising answer, and treat it as having answered the quality question. That is the demo problem from module 1 wearing an engineering hat.

Under the EU AI Act, a system classified as high-risk carries obligations that are, read closely, obligations to have run evals and kept the results. Article 15(1) requires an appropriate level of accuracy, robustness, and cybersecurity across the lifecycle. Article 15(3) requires that accuracy levels and the relevant accuracy metrics be declared in the instructions for use — you must state your metric and your number, to your deployer, in writing. Article 15(4) addresses resilience against errors and inconsistencies, which is an adversarial-slice obligation in everything but name. Article 9(6)–(8) requires testing against prior-defined metrics and probabilistic thresholds before placing on the market. And Annex IV 2(g) requires the technical documentation to describe the validation and testing procedures, the metrics used, and dated test reports signed by the responsible persons.

Every one of those is discharged by an artifact you have already built in this guide, or it is not discharged at all.

A number is not evidence. A number with foundation and chain of custody is evidence, and the difference is the whole of this section.

Foundation: what produced this number, and can it be reproduced? An unversioned “94% policy adherence” answers nothing — 94% of what set, graded by which rubric, scored by which judge, on which date, against which system version. Any of those five moving changes the number, and the first question a competent auditor asks is which set version produced it.

Chain of custody: can you show the artifact has not silently changed since? Immutable tagged dataset versions, a rubric change log, signed gate entries, and retention.

The eval record is that stack. Five layers, each of which discharges a different obligation family:

Every pathology this guide has named, consolidated as a field reference. Read the symptom column first; that is how these actually present.

Two of these deserve emphasis because they are the ones that hide the others. The single aggregate score is the pathology that makes happy-path sets, slice regressions, and dimension trades all invisible simultaneously — figure 3.1 exists to make this permanent in your memory. And passing-eval-as-proof is the one that converts every other pathology on this list from a measurement problem into a claim you have made to someone who will remember it.

Return to module 1's rule, now that everything hangs off it. An eval is an attempted refutation. A passing suite establishes that the failures you thought to attempt did not occur, on the artifacts you versioned, at the time you ran it. That is the strongest true statement available, and it is worth practising the phrasing, because the moment you need it — a launch review, an audit, an incident, a deposition — is not the moment to be composing it.

The launch-review version: “On golden set v5, rubric v2.1, judge v3, calibrated to human labels at 0.86 agreement on 24 June, the drafter met the policy-adherence threshold on 14 of 14 reported slices and the PII threshold on all cases, including 60 adversarial probes. The refund-dispute German slice sits at 0.96 against a 0.95 bar, up from 0.41 pre-implementation. We have not tested Danish multi-issue disputes, which are 2% of traffic and go to human drafting until the slice exists.”

The auditor version: the same sentence with the artifacts attached and the last clause emphasized, because the last clause is what makes the rest credible. A record that names its own gaps reads as a record kept by someone who was looking.

What you are left with, if you have built all of this, is one artifact doing four jobs at once. It is the spec — the place your team's disagreements about quality get resolved into thresholds. It is the alarm — the gate and the guardrails that catch a regression before your users write in. It is the record — the versioned, dated, signed trail that answers “how do you know.” And it is the institutional memory — every way the system has ever failed, written down as a case that will never stop being checked, long after everyone who was there has moved on.

None of that requires the suite to be finished, because it never will be. It requires only the standing habit: every failure sharpens the refutation. Ship the forty-case version this afternoon, and let production teach it the rest.