Systems That Confess

Monitoring is watching for failure modes you predicted. You knew the disk could fill, so you graphed the disk; you knew the queue could back up, so you alerted on its depth. Monitoring is necessary and it is finite: it covers exactly the failures someone thought of in advance, which is a strictly smaller set than the failures available to you.

Observability is a stronger property. A system is observable to the degree that you can answer questions about its internal state — including questions nobody anticipated — from its external outputs, without shipping new code. The clause that carries the weight is the last one. If answering "which tenants saw truncated answers between 09:20 and 09:40?" requires a deploy, you did not have that answer during the incident; you had it forty minutes after the incident, which is a different and much less valuable thing.

Observability is therefore a design property, not a product you install. Vendors sell storage and query engines for telemetry; they cannot sell you telemetry your code never emitted. The decisions that make a system observable — what constitutes an event, which identifiers travel with a request, which dimensions are worth aggregating — are made in code review, months before the incident that depends on them.

The three signals differ in what they record, what they can answer, and how their cost grows. Those three facts are related, and once you see the relation the routing rule becomes obvious rather than memorized.

Logs record individual occurrences in arbitrary detail. They answer what exactly happened in this one case. Cost grows with events × bytes, which means arbitrary high-cardinality detail — a request id, a tenant id, a corpus version, the user's own query string — is comparatively cheap to carry. What logs cannot do is answer aggregate questions cheaply: computing "p95 latency this week" from raw log lines means scanning the week.

Metrics record pre-aggregated numbers over time. They answer how much, how often, is it getting worse. Cost grows with the number of unique label combinations — every distinct pairing of dimension values is a separate time series to store and index forever. That geometry makes metrics almost free to query and unforgiving of unbounded dimensions, which is the whole of module 2's cardinality discipline in one sentence.

Traces record one request's path across services as a tree of timed operations. They answer where in the chain did the time or the failure go. Cost grows with stored spans, and because a single request may emit dozens, tracing at full volume is the signal that most reliably surprises people on the invoice. Traces are the only signal in which causation across service boundaries is a recorded fact rather than an inference from clock-skewed timestamps.

Run the rule against real questions from the compliance assistant — the retrieval-plus-tools system from Guide Nº 02, now serving roughly 10,000 queries a day for law firms answering policy questions against their own corpus.

"Why did request req_8h3k cite a policy that was repealed in 2023?" — one case, needing arbitrary detail: the log event for that request, which carries the corpus version it retrieved against, then its trace to see whether the stale document came from retrieval or from the model ignoring the retrieved set. A dashboard cannot answer this; an aggregate has already averaged away the instance you care about.

"Is p95 answer latency worse this week than last?" — an aggregate over hundreds of thousands of requests. A metric answers it in milliseconds because the aggregation already happened at write time. Answering it from logs means scanning two weeks of events, which is slow, expensive, and — if your hot retention is 14 days — sometimes impossible.

"Which dependency makes Tuesday afternoons slow?" — a question about location in the chain, asked in aggregate: trace data grouped by span, comparing Tuesday afternoon's span duration distribution against baseline. Metrics per service could approximate it, but only traces attribute a request's wall clock to the specific hop that consumed it.

Telemetry decided at write time costs a code review. Telemetry missing at incident time costs an outage-length archaeology session plus a deploy under pressure, which is the most expensive way anyone has yet invented to add a log line. The asymmetry is large enough that instrumentation belongs in the definition of done rather than in the backlog.

The standing rule the rest of this course assumes: every new endpoint ships with three things or it is not finished. One wide structured event per unit of work (module 2). Its RED metrics — rate, errors, duration — with bounded labels (module 2 again, for what "bounded" costs). Trace context propagated in and out, including across every async boundary it touches (module 3). Three artifacts, reviewable in the same pull request as the feature.

One consequence worth internalizing early: you cannot instrument your way out of an architecture that hides its own behavior. If work happens in a background thread with no context, in a third-party call with no timing, or in a queue consumer that mints fresh identifiers, no amount of log volume reconstructs the causal chain. Observability constrains design — which is the first hint of this course's thesis that reliability is a computed property of architecture, not an attitude toward it.

Here is a real logging style, lightly cleaned up:

INFO  Handling request for user
INFO  Retrieved 8 documents
WARN  Slow response from provider, retrying
ERROR Something went wrong in retrieval :(

Now the question an on-call engineer actually has at 09:20 on a bad morning: show me every failed answer for tenant acme-legal in the last hour, with its latency and which dependency failed. Against that log stream the query does not exist. There is no tenant on any line, no request identifier joining the four lines to each other, no latency, no machine-readable outcome, and the one line that reports a failure reports it in a form that cannot be counted, grouped, or alerted on.

The same request, instrumented as a structured event:

{"ts":"2026-03-17T09:20:41.882Z","service":"assist-api",
 "event":"answer_complete","req_id":"req_8h3k",
 "trace_id":"4f2a91c7","tenant":"acme-legal","tenant_tier":"enterprise",
 "outcome":"error","error_kind":"provider_timeout",
 "latency_ms":11240,"docs_retrieved":8}

Now the query is a filter and a group-by. More importantly, so is the next question, and the one after it — including questions nobody has thought of yet, which is the entire point of module 1's definition.

The discipline that gets the most out of structured logging is the wide event: one record per unit of work, emitted at completion, carrying everything known about that unit of work by the time it ends. Not one line per step — one wide record per request. This is a deliberate trade: you pay in bytes per event, and you are repaid in every join you do not have to perform, because the fields that would have needed correlating are already on the same row.

The compliance assistant's canonical answer event, in full:

{
  "ts": "2026-03-17T09:20:41.882Z",
  "service": "assist-api",
  "version": "2026.03.11-a91c",
  "event": "answer_complete",

  "req_id": "req_8h3k",
  "trace_id": "4f2a91c7",
  "session_id": "sess_71dd",

  "tenant": "acme-legal",
  "tenant_tier": "enterprise",
  "user_id": "u_2249",

  "corpus_version": "2026-03-01",
  "model_id": "primary-v4",
  "docs_retrieved": 8,
  "tools_called": ["lookup_case"],
  "tokens_in": 6120,
  "tokens_out": 1842,
  "tokens_cached": 4800,

  "latency_ms": 11240,
  "ttft_ms": 10980,
  "latency_breakdown_ms": {"retrieval": 610, "provider": 9800, "lookup_case": 420},

  "outcome": "error",
  "error_kind": "provider_timeout",
  "attempt": 2,
  "citations_valid": null
}

Four groups of fields, and the grouping is not decorative — each exists to answer a different class of incident question.

The routing question comes up in every instrumentation review: this dimension is interesting, should it be a metric label or a log field? The answer is arithmetic, and the arithmetic is unkind in one direction.

Metrics pay per unique label combination. Each distinct combination is a separate time series, stored and indexed for the full retention period whether or not anyone queries it. Suppose the assistant's latency histogram carries labels route (8 values), status (5 values), and someone adds user_id (10,000 values), with 12 histogram buckets. The series count is the product: 8 × 5 × 10,000 = 400,000 label combinations, each carrying 12 buckets plus a sum and count — on the order of five million stored series. Before the user_id label it was 8 × 5 = 40 combinations. One label multiplied the bill by ten thousand, and it did so silently, because nothing in the code review said × 10,000.

Logs pay per byte. Adding user_id to a wide event costs roughly twenty bytes on a record you were already writing — at 300,000 requests a month, about six megabytes. The same dimension is catastrophic in one signal and rounding error in the other, and the difference is entirely structural: metrics multiply dimensions, events append them.

Hence the routing rule: bounded dimensions you group by belong in metric labels; unbounded identifiers belong in events. The instinct behind the mistake is sound — you genuinely do want to slice latency by tenant — and the fix preserves it: alert and dashboard on tenant_tier, then drop into the event stream to find the specific tenant. Cheap aggregate, expensive detail, used in that order.

A correlation ID is minted once, at the outermost edge that sees the request, and then carried by every record produced on that request's behalf anywhere in the system. The compliance assistant mints req_id at API Gateway and carries it — along with the W3C trace context of module 3 — through assist-api, into the retrieval service, out to the model provider, out to case-mgmt, and across the async queue that drives conversation summarization.

That last hop is where correlation chains go to die. HTTP propagation is usually handled by a framework middleware someone installed on day one. Queue propagation almost never is: a message is published with a body and no message attributes, the consumer starts work, the logging library helpfully mints a fresh request id, and every record the consumer emits is an orphan. Nothing errors. The logs look fine. The failure surfaces only during an incident, when you follow a request into the queue and it vanishes.

Three decisions remain, and all three are made once, in the logging contract, rather than repeatedly by whoever is writing a line at the time.

Severity is a query dimension, not an emotion. The useful definition is operational: ERROR means a human or an automated policy should eventually look at this; WARN means the system handled it but the frequency is worth a graph; INFO is the normal event stream; DEBUG is off in production by default and enabled by flag, per service, for a bounded period. What makes levels break down is treating them as intensity of feeling — every author's threshold differs, and the resulting stream cannot be filtered.

Retention is tiered and stated in advance. For the assistant: hot and fully indexed for 14 days, which covers essentially all incident investigation; warm and queryable-with-latency for 90 days, which covers trend analysis and quarterly review; archived beyond that only where a legal-hold or regulatory obligation names a specific category of record. Tiering matters because undifferentiated retention is how you simultaneously pay too much and, when the audit arrives, discover you kept the wrong things too long.

Some things are never written. The standing exclusion list: credentials, API keys, and session tokens; authorization headers; full request and response bodies containing case data; and any field a redaction pass has not seen. The last item is the operative one — an exclusion list defended by good intentions fails on the first hurried pull request, so the enforcement belongs in a serializer that drops unknown fields by default rather than in a wiki page.

A span is a named operation with a start time, an end time, a span id, and a parent span id. That last field is the entire idea: it records causation. A trace is the tree of spans sharing one trace id — one request's complete life, structured.

Spans carry attributes (the model id, the number of documents retrieved, the HTTP status) and may carry timestamped events within them ("first token received", "retry 1 issued"). In practice a span's attributes and a wide event's fields converge on the same content, which is fine and increasingly deliberate: the same facts, once indexed by request for querying and once indexed by structure for timing.

What the tree gives you that correlated logs do not is containment. If the provider span ran from 09:20:32.041 to 09:20:41.841 and it is a child of the root span, then those 9.8 seconds are inside the request's wall clock and attributable to that hop — not adjacent to it, not concurrent with something else that might really be the cause. Reconstructing containment from four services' log timestamps requires clocks to agree to within the precision of your conclusions, and clocks on separate hosts do not.

The tree stays connected because each hop passes the current trace and span identity to the next. On HTTP this is the W3C context propagation standard: a traceparent header carrying version, trace id, the calling span's id, and flags.

traceparent: 00-4f2a91c7c8e34f1b9a7d2e5c81b40f6a-00f067aa0ba902b7-01
             ^  ^                                ^                ^
           ver  trace id (16 bytes)          parent span id     sampled flag

The receiving service reads the header, starts its span with that parent, and passes its own span id onward. Auto-instrumentation for common frameworks does this without anyone writing code, which is why HTTP hops rarely break — and why the hops that are not HTTP break so reliably.

Where propagation silently fails:

• Queues and event buses. Same failure as module 2's correlation ID, same fix: traceparent as a message attribute, read back and set as the consumer span's parent — usually as a link rather than a direct parent, since the consumer's work is triggered by, but not contained within, the producer's span.
• Thread pools and executors. Trace context typically lives in thread-local or task-local storage; hand work to a pool and the worker thread has an empty context. The child span becomes a new root.
• Background and scheduled jobs. There is no inbound request to inherit from, so they start fresh — correct behavior that becomes a gap when the job is really finishing a user's work.
• Hand-rolled HTTP clients. The one place in the codebase where someone used a raw socket or a bespoke wrapper is the one hop that drops the header.

The diagnostic signature is consistent and worth memorizing: orphaned spans — work that appears in the tracing backend as its own tiny root trace, with no parent, at the same moment your real trace shows an unexplained gap. Traces that look suspiciously fast are the other tell: the work happened, it just is not in this tree, and the waterfall now under-reports the request's real cost.

A waterfall lays each span on a horizontal time axis, indented by depth. Reading one is a disciplined act with a specific order: find wall-clock dominance first, then look at structure, then reach a verdict. Doing it in the other order — spotting something you know how to fix and fixing it — is how teams spend a sprint optimizing a query that accounts for five percent of the latency.

Here is the assistant's slow request from the morning of the incident.

Three verdicts a waterfall can support, and the evidence each requires:

• Parallelize — two or more child spans run end-to-start but share no data dependency. Evidence: the gold bracket in Figure 3.2. Upper bound on the gain: the shorter span's duration, never more.
• Cache — the same span with the same inputs recurs across traces, and the result is stable for a useful window. Evidence comes from comparing traces, not from one waterfall.
• Descope — a span is on the critical path and does not need to be. The lookup_case call becoming non-blocking (render the answer, populate live case data when it arrives) is a descope, and it is also what makes that dependency non-fatal in module 6's availability math.

At 10,000 requests a day and roughly a dozen spans each, full retention is about 3.6 million spans a month — survivable. At a hundred times that volume it is not, and the invoice arrives before the realization does. Sampling is therefore not a compromise you make reluctantly; it is a design decision with a right answer that depends on what you intend to ask.

Head sampling decides at the root, before the request runs: roll the dice, set the sampled flag in traceparent, and every downstream service honors it — so traces are complete or absent, never partial. It is cheap and it requires no buffering. Its defect is fatal in exactly the case you care about: the decision is made before the outcome exists, so a 1% head sample keeps 1% of your errors and 1% of your slow requests. On a rare failure you will have nothing.

Tail sampling decides after the request completes, when the outcome is known: keep every trace containing an error, every trace over a latency threshold, a small baseline of ordinary ones, and drop the rest. You get precisely the interesting population. The cost is that every span must be buffered somewhere until the verdict — a collector holding traces in memory for a decision window, which is real infrastructure with its own failure modes and its own capacity to lose data under load.

The assistant's arithmetic: 10,000 requests/day × 12 spans ≈ 120,000 spans/day at full retention. Head-sampling 10% keeps ~12,000. Errors run around 0.3% in a healthy month (~30 requests/day) and traces over 8 s about 1.5% (~150/day), so tail rules add roughly 2,200 spans — call it 12% of full volume, with 100% of the errors and 100% of the slow requests retained. During the incident, when the error rate hit 30%, that policy adapted on its own: the interesting population grew because the interesting population was what the policy was defined over.

An SLI is a ratio: good events divided by valid events, over a window. Three words in that sentence are load-bearing. Good must be machine-decidable — which is why module 2 insisted on an outcome field. Valid excludes events that were never yours to serve: malformed requests, unauthenticated probes, a client that hung up. And events, rather than time, because "minutes of downtime" is a poor proxy for a system that degrades partially and unevenly across tenants.

The part teams routinely skip is that the measurement point is part of the definition. The same nominal SLI measured in different places produces different numbers and, more importantly, different blind spots.

The compliance assistant's three SLIs, defined precisely:

Note the latency SLI is split. A streaming answer has two moments a user feels differently: when text starts appearing, and when it stops. A single number averages a fast-starting long answer against a slow-starting short one and tells you about neither. Module 8 returns to why this split is an LLM-specific instrument rather than fussiness.

An SLO is the target the SLI must meet over its window. The most common way to choose it is also the most damaging.

The right question is not "how reliable are we?" but "what does the user need, and what would they do differently if we were better?" For the compliance assistant: an associate researching a policy question tolerates an occasional failure they can retry — the work is not real-time, the alternative is reading the policy manual, and the retry costs them fifteen seconds. What they do not tolerate is an outage during a filing deadline, or answers that arrive too slowly to stay in the flow of work. That analysis points at high-but-not-heroic availability with real attention to latency and tail behavior, which is exactly what 99.5% with a hard latency SLO expresses.

Second input: the dependency math. A chain of respectable services composes to a number that may already be beneath the SLO you are considering. The assistant's chain composes to about 98.76% as originally built — module 6 does this arithmetic in full — which means 99.5% was not merely ambitious, it was arithmetically unavailable until the architecture changed. That is the discipline this course is arguing for: the SLO is a computed consequence of what you built, and if you want a better number you buy it with a design change, not with an intention.

The downtime table makes the stakes concrete. On a 30-day month of 43,200 minutes:

Read the third column across and the honest negotiation becomes possible. Moving from 99.5% to 99.9% is not a 0.4% improvement; it is a commitment to absorb five times less failure, which converts into architecture (a second provider, a degraded mode, multi-AZ everything) and into operational cost (an on-call rotation that can respond inside 43 minutes a month, cumulative). The answer to "why not 99.9%?" is a price, and the price is legible.

An SLO is an internal target. An SLA is a contractual commitment with remedies attached — service credits, termination rights, sometimes indemnities. They are different instruments with different audiences and, critically, different numbers.

The discipline is to set the SLO tighter than the SLA, so that the contract never triggers first. If the assistant's SLA promises 99.0% and the internal SLO is 99.5%, then the internal target is breached — and the error-budget policy engages, and engineering attention is redirected — while the contract is still comfortably satisfied. The gap is your warning track.

Which is why "just publish our SLO" is a bad trade even when the number is achievable. It converts an engineering target — revisable quarterly as you learn what users need — into a contractual floor that requires a negotiation to change. Internal targets should move. Contracts should not.

Now the instrument. If the SLO is 99.5% over 30 days and you serve 300,000 requests a month, the error budget is 0.5% — 1,500 bad requests, or 216 minutes of full outage equivalent. That is not a shameful remainder. It is permission, purchased in advance, and it is the thing that lets an engineering organization move.

Things worth spending budget on: shipping a risky-but-valuable change behind a fast rollback; migrating the retrieval index during business hours instead of at 2 a.m., where the on-call is alert and mistakes are cheaper; running a deliberate failover exercise; letting a canary bake in production with real traffic rather than simulating forever. Each has a cost in expected bad requests, and each is a choice you can now make explicitly.

The error-budget policy is the pre-agreed rule for what happens at each state. Agree it when everyone is calm; its whole value is being decided before the argument.

The corollary is uncomfortable for cautious teams: an unspent budget is waste. If you end three consecutive months at 99.95% against a 99.5% objective, you did not win — you paid for risk capacity and did not use it, which means either you were too conservative with your roadmap or the SLO is set below what your architecture now delivers and should be re-examined. Budgets are for spending.

Availability and latency have a convenient property: the verdict on each event is free and instant. A status code exists; a clock exists. Citation validity has neither. Whether an answer correctly cited a policy that supports it is a judgment, produced by the eval pipeline of Guide Nº 03 running a validator over a sample of answers — and that changes four things about how the SLI behaves.

It is sampled, so it has error bars. At 200 graded answers a day, an observed 97% could be a true 97%, or a true 98% having a mediocre day; the standard error on a 98% rate at n = 200 is about one percentage point, so single-day movements of a point or two are noise. Treating them as signal produces a team that chases ghosts.

It is slow. Detecting a real one-point drop with confidence takes days of accumulated samples, not minutes. This is not a defect to be engineered away — it is the nature of measuring a distribution rather than an event.

It therefore never pages. Module 5's routing rule falls out of the arithmetic rather than from a preference: a signal that cannot distinguish a real regression from noise inside a few hours cannot justify waking anyone. Quality regressions get a ticket with a named owner and a deadline.

Its window is shorter. Citation validity uses a rolling 7 days rather than 30, because a corpus update or a silent model change can shift quality abruptly and a 30-day window would dilute a fresh regression under three weeks of good history.

What does not change is the governance. Citation validity has an SLO (≥ 98% over 7 days), an error budget (2% of graded answers), and a policy: sustained breach blocks prompt and retrieval changes until the regression is understood, on the same logic that a burned availability budget blocks infrastructure migrations. Same instrument, different sensor and different clock.

The rule is one sentence: page on what users feel; ticket what might explain it. A symptom-based alert fires on an SLI degrading — availability falling, latency exceeding its threshold. A cause-based alert fires on an internal condition that might produce that, and might not.

Three real cause-alerts from the assistant's stack, rewritten:

Notice what the rewrite preserves. The cause signals do not disappear — they are exactly what the responder needs in the first ninety seconds of an investigation. What changes is their authority to wake someone. A signal can be indispensable for diagnosis and still have no business interrupting a human at 2 a.m.

"Error rate above 0.5%" is a bad alert, and it is bad in an instructive way: 0.5% is the budget, and a budget is a monthly quantity, so an instantaneous comparison against it is a category error. Sustained 0.5% failure spends exactly the budget over 30 days — nothing to page about. A 30% error rate for six minutes is a genuine emergency. One threshold cannot express both.

Burn rate fixes this by normalizing: it is how many times faster than sustainable the budget is being consumed. Burn rate 1 means you will spend exactly the budget over the window. Burn rate 60 means you will spend the whole month's budget in 12 hours.

burn rate = observed error rate / error budget rate

  0.5% errors  ÷ 0.5% budget =   1×   sustainable
  7.2% errors  ÷ 0.5% budget =  14.4× fast burn
 30.0% errors  ÷ 0.5% budget =  60×   the incident

Now the thresholds derive from a decision that is actually meaningful: what fraction of the month's budget am I willing to lose before a human intervenes? A burn rate of B sustained for a window W consumes B × W / 30 days of the budget. Choose the fraction, solve for the rate:

Check the arithmetic on the first row: 14.4 × 1 hour = 14.4 hours of budget-equivalent, and 14.4 / 720 hours in a 30-day month = 2%. The number is not folklore; it is the answer to "page me before I lose 2% of the month."

The short window is the second half of a multiwindow alert, and it exists to solve a specific annoyance: an alert on a 1-hour window keeps firing for up to an hour after the problem resolves, because the window still contains the bad minutes. Requiring the short window to also be burning means the alert says "this was severe over the last hour and it is still happening right now." Blips that self-heal in four minutes never page, and resolved incidents stop paging promptly.

Every page spends three things: the responder's attention now, their sleep tonight, and their trust in the pager permanently. The first two recover. The third does not recover easily, and its loss is the most dangerous failure in an on-call system, because a responder who has learned that pages are usually nothing will treat the real one as usually nothing.

A page is legitimate only if it is all three of the following. Urgent — it requires action within minutes, not by Thursday. Actionable — the person paged can do something; a page that only informs is a notification wearing a costume. User-visible — it corresponds to something a user is experiencing or imminently will. Fail any one and the alert belongs elsewhere.

The hygiene practice that keeps this honest is a weekly alert review, and it needs exactly one question per page: did a human take an action they would not otherwise have taken? If the answer is no — it self-resolved, it was investigated and found benign, the runbook said wait — that alert is demoted this week, not eventually. The default direction of travel is downward, because alerts accumulate by ratchet otherwise: everyone can add one, nobody feels authorized to remove one.

The comparison that ends the argument: an alert's detection latency is a configured window, typically measured in minutes and identical at 4 a.m. and 4 p.m. A dashboard's detection latency is the interval until someone looks, which on a weekend is not measured in minutes and is not measured at all.

Every candidate alert gets routed by three questions asked in order, and the order matters — a signal can pass the first two and still fail the third, which is the case people find hardest to accept.

The assistant's complete alert specification:

Seven alerts. Four page, three ticket. That is the whole surface area, and its smallness is the point — every addition should have to argue against the responder's finite attention.

The citation-validity route is the instructive exception. It is genuinely user-visible: an answer citing a repealed policy is worse for the user than a slow answer. It nevertheless routes to a ticket, because module 4's sampling arithmetic says the signal cannot distinguish a real regression from noise inside a few hours. Waking someone for a signal that cannot yet be trusted is how you get a 3 a.m. investigation that concludes "probably noise" — twice — after which nobody responds to the third one, which is real. The exception proves the rule rather than breaking it: the constraint is that pages must be actionable now, and a signal without confidence is not.

Take three services, each available 99% of the time. Wire them so that a request needs all three — a hard serial composition — and the system's availability is the product:

0.99 × 0.99 × 0.99 = 0.970299  →  97.03%

Three "two nines" services compose to worse than two nines:
1% + 1% + 1% of failure, near enough, because failures are rare
and overlaps are negligible. Unavailability adds; availability multiplies.

Now wire the same three as alternatives — any one suffices, with failover — and you get parallel composition, where the failure probabilities multiply:

1 − (0.01 × 0.01 × 0.01) = 1 − 0.000001  →  99.9999%

Three "two nines" services compose to six nines.

The practical asymmetry: removing a serial dependency is usually cheaper than making one more reliable. Going from 99.5% to 99.9% on a service you own is a quarter of engineering work. Making that service non-fatal — degrade gracefully when it is gone — often takes a sprint and removes its term from the product entirely.

Apply it to the real chain. The compliance assistant's request path traverses five components, each with an honest modeling number for its availability:

CloudFront + API Gateway     0.9995
assist-api (Flask service)   0.999
retrieval + Postgres         0.999
primary model provider       0.995
case-mgmt API                0.995

0.9995 × 0.999 × 0.999 × 0.995 × 0.995 = 0.98755…
                                       ≈ 98.76%

98.76% is roughly 536 minutes of expected unavailability per 30-day month — just under nine hours. The SLO in module 4 is 99.5%, which permits 216 minutes. The system as built could not meet its objective, and no amount of care, on-call discipline, or vigilance would have changed that: the number was arithmetically unavailable from the day the architecture was drawn.

Notice where the damage is concentrated. The two 99.5% dependencies contribute about 1.0 percentage point of the 1.24 points of unavailability between them. The two services the team owns and worries about most contribute 0.2 points combined. Effort follows attention rather than arithmetic unless someone does this calculation.

The redesign. Two changes, each targeting a 99.5% term:

Change one — make case-mgmt non-fatal. The lookup_case tool enriches an answer with live case status. When it fails, the answer is still correct and useful; it simply lacks the live enrichment. So the call gets a 1.5 s timeout and a degraded path: return the answer, flag it as "case data unavailable," carry on. The dependency's term leaves the product entirely — not improved, removed. This is the descope verdict from module 3's waterfall reading, arriving here as availability.

Change two — add a secondary model provider. Two independent providers at 99.5% each, with failover on timeout or error: 1 − (0.005 × 0.005) = 0.999975. The model dependency goes from 0.995 to ≈ 0.99998, and the honest asterisk is that both providers share the public internet and our egress path, so the realized number is somewhat below the arithmetic.

0.9995 × 0.999 × 0.999 × 0.999975 = 0.99748…
                                  ≈ 99.75%

Ceiling moves from 98.76% to 99.75%.
Expected unavailability: 536 min/month → 109 min/month.
The 99.5% SLO (216 min) now has ~107 minutes of headroom
for everything the model doesn't capture: deploys, config
errors, human mistakes, correlated failures.

99.999% availability is 26 seconds of unavailability per 30-day month. Sit with that number before agreeing to it, because it forecloses things people rarely think to check.

Humans are excluded by arithmetic. If a page reaches a responder in 60 seconds and they take 90 seconds to orient, the acknowledgement alone has consumed six months of budget. Five nines means detection and remediation are automated end to end; the human role is reviewing what the automation already did.

Your dependencies compose below it. A single 99.99% dependency in the serial path caps you at 99.99%, an order of magnitude short, before you have made a single mistake of your own. Five nines requires every serial dependency to be five nines or better, or to have been made non-fatal.

Measurement noise swamps it. At 300,000 requests a month, 26 seconds of budget is roughly three failed requests. Three. Your measurement pipeline has more error than that — a load balancer log line lost in shipping, a synthetic probe flapping on its own DNS, one client that hung up in a tunnel. You would be reporting on the noise floor of your own telemetry.

So when an executive asks for five nines, the useful response is neither agreement nor refusal. It is a counter-offer with the arithmetic attached: "Five nines is 26 seconds a month, which excludes any human from the loop and requires every dependency at that tier — including the model provider, whose observed availability is around 99.5%. What we can commit to is 99.9% on the answer path with a documented degraded mode, which costs roughly one quarter of engineering and $14k a month in redundancy. If some specific operation genuinely needs more, name it and we will price that operation alone." That last clause is where the conversation usually resolves, because the request is almost never about all operations — it is about one flow somebody is worried about.

Every team eventually discovers that a system running comfortably at 70% capacity becomes unusable at 95%, and that the transition is not gradual. The shape behind this is worth carrying around as intuition, and one formula supplies it: for a simple single-server queue, mean time in system ≈ service time / (1 − utilization).

ρ = 0.50  →  1/(1−0.50) =   2×  service time
ρ = 0.70  →  1/(1−0.70) = 3.3×
ρ = 0.90  →  1/(1−0.90) =  10×
ρ = 0.95  →  1/(1−0.95) =  20×
ρ = 0.99  →  1/(1−0.99) = 100×

Between 50% and 70% utilization, latency less than doubles. Between 90% and 99% — an increase of nine percentage points, the kind of change a capacity plan describes as a rounding error — latency grows tenfold. That asymmetry is the whole lesson: the cost of load is not linear in load, and the interval in which the system tells you it is unhappy is short.

Three operational consequences follow directly.

Headroom is what your latency SLO costs. Running at 50% rather than 90% doubles your compute bill and buys a 5× improvement in queueing delay. That is not waste; it is the purchase price of the latency number you committed to in module 4. If someone proposes cutting it, the honest framing is that they are proposing to spend latency budget on infrastructure savings — a legitimate trade, made explicit.

Autoscaling targets near 90% are a cliff, not an optimization. Scaling takes time — instance start, warm-up, health checks, connection draining — during which traffic keeps arriving. Starting the scale-up at 90% means the curve is already steep when the new capacity begins its two-minute journey. A target between 60% and 70% keeps the system on the flat part while capacity catches up.

The metric to watch is not the mean. The formula describes mean time in system; the tail is worse and steeper. A system whose p50 has doubled has usually already lost its p99 entirely, which is why the latency SLI in module 4 is a percentile.

Now put the two curves together, because the incident that runs through this course lives at their intersection.

At 09:10 the primary model provider's p95 time-to-first-token begins drifting from 1.4 s toward 11 s. Nothing has failed. The provider is slow, which by itself would be a latency-SLO problem and not an availability one. Then the client timeout — 2 seconds, chosen years ago to protect the user experience — converts every slow response into an error. And the client, correctly by its own lights, retries. Three attempts each.

Offered load, minute by minute:

  baseline           30 req/min → 30 provider calls/min
  slow, 1st timeout  30 req/min → 60 calls/min  (each request now 2)
  full retry policy  30 req/min → 120 calls/min (3 attempts + original)

Provider capacity meanwhile: effectively halved by its own degradation.
Offered load ×4, capacity ×0.5 → utilization ×8 against the curve above.

This is a retry storm. Its defining property is that it is self-reinforcing: retries raise utilization, higher utilization raises latency, higher latency fires more timeouts, more timeouts produce more retries. The system does not drift into this state — it accelerates into it, which is why the transition from "slightly slow" to "completely down" took four minutes.

The four countermeasures, and where each operates on Figure 6.2:

Jitter deserves its own sentence, because backoff without it is a common half-measure. If a thousand clients all time out at the same moment and all wait exactly one second, they retry at the same moment. Backoff alone reschedules the thundering herd; it does not disperse it. Randomizing each client's delay uniformly over the interval is what converts the wave into a flow.

Severity is a claim about user impact and budget burn — not about how embarrassing the cause is, how senior the person who caused it was, or how confident anyone is about what is happening. The last point is the one teams get wrong: severity is assigned from the symptom, before the cause is known. If you wait to understand the problem before declaring, you have made declaration a reward for having already done the work.

The morning of the incident: the fast-burn alert fires at 09:14 with an observed error rate near 30% — a burn rate around 60×. That is a SEV2 by the table, and it is declared at 09:19 on the burn rate alone. Nobody knows the cause. Declaring anyway is correct, and it is the single highest-leverage decision of the hour.

Teams under-declare for reasons that are social rather than technical: declaring feels like admitting you cannot handle it, and there is a persistent belief that the cost is measured in disruption. The counter is structural — make declaration a routine act with a low ceremony floor, praise it in review even when the incident turns out to be nothing, and put the burn-rate threshold in the severity table so the decision is mostly arithmetic.

Three roles, and the discipline is that one person holds one of them.

Incident commander. Owns the incident: assigns work, makes calls, decides when to mitigate versus keep investigating, decides when it is over. Incident commander is explicitly not a debugging role. The moment the commander opens a terminal, the coordination function stops — and coordination is what the incident is short of, because investigation is naturally absorbing and someone has to keep track of who is doing what, what has been ruled out, and whether it is time to stop diagnosing and start mitigating.

Ops lead. Hands on the system. Runs the queries, reads the traces, executes the mitigations. Reports findings to the commander rather than acting unilaterally, so the commander's model of the incident stays current.

Comms lead. Stakeholder updates on the cadence the severity table requires. This role exists because otherwise the most senior person in the room does it — badly, between queries, while being the person everyone is waiting on.

Comms discipline: one channel, timestamps, facts not speculation. One channel because a distributed conversation cannot be reconstructed. Timestamps because the postmortem's timeline is assembled from this record. Facts not speculation because of what the record becomes later.

The full run. Every number here is consistent with the burn chart of Figure 5.1 and the arithmetic of module 6.

09:10 — Onset. The primary model provider's p95 time-to-first-token begins climbing from 1.4 s. No alert; nothing has crossed a threshold yet.

09:12 — TTFT passes the 2 s client timeout. Slow responses start becoming errors. Retries begin, quadrupling offered load to the provider (module 6's amplification).

09:14 — Fast-burn alert fires. Error rate ≈ 30%, burn rate ≈ 60×.

ALERT  availability-fast-burn  SEV-candidate
service   assist-api
sli       availability (LB, schema-valid 200s)
burn_rate 59.8x  (windows: 1h ✓  5m ✓)
budget    93.1% remaining → projected exhaustion in 12.0h
top_error provider_timeout  (94% of failures)
runbook   go/assist-runbook#failover

The alert payload is doing real work: it names the dominant error kind, so the responder starts with a hypothesis rather than a blank page.

09:19 — SEV2 declared on burn rate alone; roles assigned; channel opened. Cause unknown, and declaring anyway is the correct call.

09:23 — Ops lead confirms blast radius from the SLI: all tenants, all tiers, ~30% of requests. Not a single-tenant problem, which rules out corpus and tenant-config hypotheses in one query.

09:27 — Trace waterfall isolates the provider span (the read performed in Figure 3.2: 9.8 s of an 11.2 s request). The wide events agree:

{"ts":"2026-03-17T09:26:58.114Z","service":"assist-api",
 "event":"answer_complete","req_id":"req_8h3k","trace_id":"4f2a91c7",
 "tenant_tier":"enterprise","model_id":"primary-v4",
 "latency_ms":11240,"ttft_ms":10980,
 "latency_breakdown_ms":{"retrieval":610,"provider":9800,"lookup_case":420},
 "outcome":"error","error_kind":"provider_timeout","attempt":2}

09:31 — Retry amplification recognized. Outbound provider call rate is ~4× inbound request rate; the team is now part of the provider's problem. This recognition took seventeen minutes and should have taken two — attempt was on the event, but outbound call rate was not instrumented separately from inbound, so the amplification had to be inferred by comparing two dashboards by eye.

09:36 — Commander sets a decision clock: "failover by 09:45 unless the provider's own status changes." This is the most underrated move in incident management — a pre-committed time at which investigation stops and mitigation begins, decided while everyone is still calm enough to commit to it.

09:41 — Mitigation. Two actions: fail over to the secondary provider, and disable the non-essential re-ranking stage to cut per-request work. Error rate falls below 2% within three minutes.

{"ts":"2026-03-17T09:41:07.402Z","service":"assist-api",
 "event":"config_change","actor":"ops-lead","incident":"INC-2026-0317",
 "change":"model_provider.primary→secondary","scope":"all_tenants",
 "reason":"provider_timeout burn 60x"}

10:05 — Resolved. SLI stable for 20 minutes; re-ranking restored; incident closed with the failover left in place pending the provider's own resolution.

The intervals — the numbers a postmortem is actually graded on: detection 4 min (09:10 → 09:14, excellent, and entirely attributable to burn-rate alerting on a symptom); declaration 5 min (09:14 → 09:19); diagnosis 13 min (09:14 → 09:27, tracing earning its cost); mitigation 27 min (09:14 → 09:41, the interval with the most room in it); resolution 51 min total. Budget consumed ≈ 60× × 51 min ÷ 43,200 min ≈ 7.1% of the month — about 105 failed requests of the 1,500-request budget's 7%.

A blameless postmortem assumes competent people operating in a system that made the failure easy. This is a methodological commitment rather than a kindness: the moment attribution is on the table, the people with the most information have the strongest incentive to shade it, and you lose access to the only account of what actually happened. Blamelessness is how you keep the report accurate.

It follows that you look for contributing causes, plural, rather than a root cause, singular. Real incidents are conjunctions. This one required: a 2 s client timeout calibrated against normal provider latency with no margin; no retry budget, so amplification was unbounded; no automatic provider failover, so mitigation was a human decision requiring a human to be convinced; and no outbound-call-rate metric, so the amplification took seventeen minutes to see. Remove any one and the incident is smaller. Naming one of them "the root cause" is a choice about where to stop looking, and it is usually made at whichever cause has a person attached.

The document skeleton — six sections, and the impact section stated in budget terms rather than adjectives:

## INC-2026-0317 — provider latency spike
Owner:    Status: draft | in review | final

### 1. Summary
Two or three sentences. What broke, for whom, how long,
how it was mitigated. Readable by someone with no context.

### 2. Impact
Users affected, requests failed, budget consumed (% and events),
SLO status after. Numbers, not adjectives.

### 3. Timeline
Timestamped, from onset to resolution. Include detection,
declaration, diagnosis, mitigation. Facts only.

### 4. Contributing causes
Plural. Each with the evidence that supports it.
No person is a cause.

### 5. What went well
Not morale decoration — these are the controls that worked
and must not be traded away in the fixes.

### 6. Action items
| # | Action | System property changed | Owner | Due | P |
Every row changes a system property. No vigilance items.

Section 5 earns its place: without it, the fixes proposed in section 6 sometimes undo the controls that limited the damage. Here, burn-rate alerting caught the incident in four minutes and tail-sampled traces made diagnosis take thirteen. If a later cost review proposes cutting tracing, section 5 is the record of what that cut would cost.

Cost varies per request by orders of magnitude. An ordinary HTTP endpoint costs approximately the same to serve every time; capacity planning is about request count. A model call's cost is a function of tokens in and tokens out, and across the assistant's traffic that ranges from about 900 tokens for a short lookup to 14,000 for a long comparison across many retrieved documents. Request count no longer predicts spend, so cost has to become a measured dimension rather than an inferred one.

Latency is two numbers, not one. A streaming answer has time-to-first-token — how long the user stares at nothing — and time-to-complete, which is dominated by output length. These move independently and for different reasons: TTFT reflects the provider's queue and the prompt's size, completion time reflects how much the model decided to say. Averaging them produces a number describing no experience anyone had. Module 4's split SLI is the consequence.

Correctness is a distribution, not a code. A 200 response tells you the pipeline worked. Whether the answer cited a policy that actually supports the proposition is a judgment, produced by a grader over a sample, arriving hours later with error bars. And the dependency can change its behavior without any version bump you can observe — the same model id, the same prompt, quietly different outputs.

Token counts become first-class metrics, dimensioned by model_id, feature, and tenant_tier. Note the last one — tier, not tenant id. This is module 2's cardinality rule arriving in a place where the temptation to violate it is strongest, because per-tenant cost attribution is a thing finance genuinely wants. The resolution is the same as before: bounded dimensions in metrics for alerting and trend, the tenant id on the wide event for the drill-down, and a nightly batch job over the event stream for the per-tenant cost report finance actually needs monthly rather than in real time.

The assistant's numbers, per answer, averaged over a healthy week:

Alert on cost per answer, not on total spend. This is the single most useful design decision in cost telemetry, and it is not obvious. Total spend rises when traffic rises, which is good news; it also rises when a prompt change adds 3,000 tokens of instructions to every request, which is not. A total-spend alert cannot distinguish them, so it either fires on growth (and gets muted) or is set high enough to miss a 40% per-unit regression hiding inside a good month. Cost per answer, dimensioned by feature, isolates the regression: traffic doubling leaves it flat, and a prompt regression moves it immediately.

Latency percentiles, per model. A provider serves from a heterogeneous fleet, and providers roll out new serving stacks progressively. Aggregate p50 hides the slow fleet entirely; aggregate p95 shows something is wrong without saying what. Percentiles dimensioned by model_id — and, where the provider exposes it, by region or fleet id — turn "latency is worse" into "the p95 on primary-v4 is worse and primary-v3 is unchanged," which is a sentence you can act on within the hour.

Segment completion latency by output-length band as well. A model that has started producing longer answers will show rising p99 time-to-complete while nothing about the system got slower — the answers got longer. That is a product change disguised as a performance regression, and telling them apart requires the segmentation.

The failure this section exists for: every status code is 200, latency is nominal, availability is green, and the answers have quietly become worse. Nothing in modules 1 through 7 detects this, because every instrument built so far measures whether the machinery ran.

Quality drift has three usual causes. A silent model change — the provider updates the weights behind a stable model id, and outputs shift. Corpus rot — documents are added, superseded, or reindexed, and retrieval starts returning subtly wrong context. And prompt entropy — a series of individually reasonable prompt edits accumulate into instructions that pull against each other.

Four signals, in decreasing order of directness and increasing order of cheapness:

Citation validity from sampled eval grading — the primary quality SLI, 200 answers a day graded by the validator pipeline of Guide Nº 03. Directly measures the thing you care about, and costs a real grading run.

Refusal rate — the fraction of answers declining to answer. Cheap, computed from the answer itself, and a superb early-warning tripwire: a model update that made the model more cautious shows up here within hours, long before enough graded samples accumulate to move the citation SLI.

Answer-length distribution — nearly free, and a surprisingly sensitive change detector. A shift in mean answer length is not itself a quality verdict, but it is strong evidence that something about generation changed, which is exactly the alert you want when the provider will not tell you.

Retrieval hit rate — the fraction of answers citing at least one retrieved document, and the mean rank of cited documents. Isolates corpus rot from model drift, which is the first fork in any quality investigation.

Which is why the drift protocol is written in terms of the cheap signals first: refusal rate and answer-length distribution alert on a 24-hour window because they have thousands of samples a day; citation validity confirms over 7 days because it has 200. The cheap signals say something changed; the expensive one says and it is worse. Wiring them in that order is what gets you from a silent model change to a confirmed regression in a day instead of a week.

Here is the tension, stated without softening. The single most useful artifact for debugging a quality problem is the full prompt and the full response: the user's actual question, the retrieved documents, the model's actual words. For the compliance assistant those contain client names, matter details, and a lawyer's own characterization of a live case — personal data, some of it special-category, all of it professionally sensitive. It is simultaneously the thing you most need and the thing you least want in a general-purpose log store with 90-day retention and organization-wide read access.

The resolution is that "capture" is not one decision. It is four decisions about four classes of telemetry, each with its own purpose, sampling rate, redaction, retention, and access control.

Three design choices in that table carry most of the weight.

Redaction happens at ingest, not at query time. If unredacted text lands in storage and is masked on the way out, you have stored the personal data and built a permission system — one misconfiguration away from exposure, and no help at all against a subject access request that asks what you hold. Redacting before persistence means the data was never collected in that form.

Full capture is a flag, and the flag has an expiry. The failure mode is not turning it on; it is leaving it on. Bind it to an incident id, default it to a seven-day auto-delete, and provide no extension path — if you still need it on day eight, that is a new decision with a new justification, which is exactly the friction you want.

Access to full captures is logged. Not to distrust engineers, but because "who accessed this?" is a question you will be asked and should be able to answer in one query. An access log is also what makes a narrow permission grant credible to a reviewer.