Later, Not Never

Start with the wine-cellar product's price-alert feature, which is the running example for this entire course. A merchant feed updates a price; users who watch that wine at that threshold should get an email. In the first version, the whole thing lives inside one HTTP request: the feed webhook arrives, the handler matches alerts, calls the email provider, writes the price history row, and returns 200.

This works beautifully until any single part of it doesn't. Four failures are usually catalogued separately — slow work, a flaky downstream, a traffic spike, and fan-out to several consumers — but they are one failure wearing four costumes. In request/response, the caller's outcome is a function of every callee's outcome. You inherit their latency, their error rate, and their capacity ceiling, all at once, at the exact moment a user is watching a spinner.

The wine-cellar case makes the arithmetic concrete. Alert matching is fast: a keyed query, 300 ms at p99. The email provider is not: it is generally 400 ms, but once or twice a week it enters an eight-second brownout where requests hang and then fail. In the synchronous design, a brownout in a third-party mail API becomes a 502 on your feed-intake endpoint — and because merchants retry failed webhooks, it also becomes a duplicate price row when the retry lands after your slow write partially succeeded.

A queue does not make anything faster. What it does is break the chain between producer and consumer along three separable axes, and being precise about which axis you need is the difference between a justified design and cargo cult.

Time. The consumer need not be running right now. If the notification service is redeploying, the price-change event waits in the broker and is processed on the other side of the deploy. Load. The queue absorbs arrival rates the consumer cannot match instantaneously; the backlog is the shock absorber. Failure. The producer's success no longer depends on the consumer's success — the merchant's webhook returns 202 whether or not the email provider is having a bad afternoon.

Load leveling deserves particular care, because it is the axis people most often misread. When a large merchant drops a nightly catalogue refresh, the wine-cellar pipeline sees roughly 50× its baseline publish rate for about four minutes. The queue swallows it. But the queue did not add capacity — it borrowed time. Every message in that spike still has to be processed by the same consumers at the same rate, and the backlog drains only because the arrival rate eventually falls below the processing rate.

Here is the whole trade in one line. Synchronously, the response means the work happened. Asynchronously, the response means the broker has the message. Those are different promises, and every remaining module of this course exists to buy back some named fraction of the difference.

It is worth reading the syllabus as a price list against that surrendered certainty. Acknowledgments buy back the broker has it durably (m2). At-least-once delivery buys back it will be processed (m3). Idempotent consumers buy back it will be processed once (m3). Dead-letter queues buy back failure will be visible (m5). Partition keys buy back related work happens in order (m6). Sagas buy back multi-step work ends in a coherent state (m7). Gauges and span links buy back I can see how far behind the truth I am (m8). No single mechanism restores the synchronous guarantee, and any vendor claiming otherwise is selling you the illusion rather than the machinery that produces it.

Asynchrony is a second system to operate, a lag to explain to users, and a substantially harder debugging story. Those costs are real and recurring, which means the case for a queue must be made per workflow rather than adopted as a house style.

The bright-line rule: if the caller needs the answer in order to decide what to do next, the work stays synchronous. Authorization decisions, validation of the caller's own input, price quotes shown before a purchase, and uniqueness checks that gate a form submission are all answers, not tasks. Queueing them means either inventing a polling protocol that reproduces request/response badly, or shipping a user experience that says “we'll tell you later whether your card was declined.”

In the wine-cellar pipeline, the split falls cleanly. Feed-credential validation is synchronous: the merchant's request is rejected in-band with a 401, because the merchant must know now whether to fix its credentials. Alert fan-out is asynchronous: nobody's next action depends on it, and its dependencies are exactly the slow, flaky, spiky ones described above.

Three components, three promises. The producer promises durable handoff: the publish call does not return success until the broker has committed the message. The broker promises durable custody: once accepted, the message survives broker restarts and is offered to a consumer until someone acknowledges it. The consumer promises processing and acknowledgment: it does the work, then tells the broker the message can be deleted.

Every queue feature you will ever configure is one of these three defending its promise against the others' failures. Publish confirms exist because producers cannot trust the network. Durability and replication exist because brokers crash. Visibility timeouts, receive counts, and dead-letter queues all exist because consumers crash, hang, or receive work they can never complete. Read a broker's configuration page with that frame and the options stop looking arbitrary.

Here is the canonical event this course uses everywhere. Every diagram, quiz, and worksheet from here forward refers to this exact payload.

{
  "event_id":      "evt_9f31",
  "type":          "wine.price_changed",
  "occurred_at":   "2026-03-14T09:12:04Z",
  "wine_id":       "wn_4417",
  "wine_name":     "Ch. Montrose 2016",
  "merchant_id":   "mch_082",
  "old_price_usd": 189.00,
  "new_price_usd": 154.00,
  "currency":      "USD"
}

Its life runs through five states, and this vocabulary is fixed for the rest of the course. Published: the producer's call has returned success and the broker has committed the message durably. Available: it is eligible for delivery and any polling consumer may receive it. In-flight: a consumer has received it, and the broker has hidden it from every other consumer for the length of the visibility timeout. Acked: the consumer confirmed completion and the broker deleted it. Two unhappy states complete the picture: redelivered — the message returned to available because the consumer nacked it or the visibility timeout expired — and dead-lettered, where a message that has been received too many times is moved off the main queue for triage.

In-flight means invisible, not gone. When a consumer receives evt_9f31, the broker starts a clock — the visibility timeout — during which no other consumer can see the message. If the ack arrives before the clock expires, the message is deleted. If it does not, the broker assumes the consumer died and returns the message to available.

That assumption is the broker's dead-man switch, and it is correct far more often than it is wrong. But notice what the broker actually observes: silence. It cannot distinguish a consumer that crashed from a consumer that is alive, healthy, and still working. A merely slow consumer is therefore the quietest source of duplicate processing in any queue system — no error is logged, no alarm fires, and two consumers do the same work in parallel while both believe they hold the message exclusively.

Everything above converges on one line of code: where you call ack. It is usually written without thought, and it decides the delivery guarantee your entire system runs on.

Consider the price-history consumer, which appends a row to the time series that powers the wine's price chart. Two orderings are possible. Ack first, then work: the message is deleted the moment it is received, and if the append fails — a transient database error, a pod eviction — nothing redelivers it. The chart has a silent gap that no alarm will ever mention, because from the broker's perspective the message was handled. Work first, then ack: the append happens, and only then is the message deleted. A crash between the append and the ack causes redelivery, and the row is appended twice.

Notice too that ack-after-work sets up a second question the code cannot dodge: what counts as “the work”? For the notification consumer, the side effect lives in a third-party email API you cannot roll back. For the price-history consumer, it is a row in your own Postgres, which you can wrap in a transaction. That difference determines how expensive deduplication will be — which is precisely where module 3 begins.

At-most-once processing: the consumer acks before doing the work. Every message is processed zero or one times. Failures lose work, and they lose it silently, because from the broker's perspective nothing went wrong. At-least-once processing: the consumer acks after doing the work. Every message is processed one or more times. Failures duplicate work, visibly and in principle detectably.

These are not two settings on a dial with a better option in the middle. They are the two sides of one unavoidable window: between the side effect and the ack, something can crash, and you choose in advance whether that crash costs you a lost message or a repeated one.

Given a choice between losing work and repeating it, almost every production system should choose repetition, for one asymmetry: duplication is detectable and suppressible; silent loss is neither. A duplicate leaves evidence — two rows, two emails, two log lines with the same event id — and can be caught by a mechanism you control. A message lost between the ack and the crash leaves nothing behind. There is no query that returns the alerts you failed to send.

For the wine-cellar pipeline the choice is not close. A duplicate price-history row is a visible artifact you can dedupe at query time; a missing one is an invisible hole in a chart that a user might notice months later, if ever. A duplicate alert email is an annoyance; a missing alert is the feature not working, silently, for the user who cared most.

But choosing at-least-once means accepting an obligation. At the wine-cellar's volume — roughly 90,000 price events a day across the three consumers — a duplication rate of even 0.05% is about 135 duplicated processings per day. Deploys alone guarantee a cluster of them, because every rolling restart evicts consumers mid-work. Duplicates are not an edge case you hope against; they are a daily quantity you engineer for.

Guide Nº 07 solved the producer's version of this problem: the client attaches an idempotency key, the server stores it transactionally with the side effect, and a resend replays the stored response instead of re-executing. The consumer's version is the same discipline pointed the other way, and it deserves the same precision — a dedup state whose location and lifetime are stated as numbers.

The idempotent consumer pattern, concretely. The dedup identity is the event's event_id — evt_9f31 — which the producer mints once per real-world occurrence and which survives every redelivery unchanged. The ledger is a table:

CREATE TABLE processed_messages (
  consumer_name  text        NOT NULL,
  event_id       text        NOT NULL,
  processed_at   timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer_name, event_id)
);

Three decisions do the work, and each has a wrong answer that looks reasonable. Scope: the key is (consumer_name, event_id), not event_id alone — price-history and notification-delivery both legitimately process evt_9f31, and a shared ledger would let whichever ran first suppress the other. Atomicity: the insert must share a transaction with the side effect. Retention: seven days, chosen as arithmetic rather than instinct. In-queue redelivery is bounded at max receive count 5 × a 30-second visibility timeout — under three minutes. The long tail is dead-letter redrive: the DLQ retains for 14 days, but the triage runbook in m5 commits to deciding within five business days, so seven days covers every redrive this system performs as routine. Anything redriven later is reprocessed deliberately, by an operator who has been told so. A nightly job deletes older rows.

The consumer body is then unremarkable, which is the point:

def handle(msg):
    with db.transaction() as tx:
        try:
            tx.execute(
              "INSERT INTO processed_messages (consumer_name, event_id) VALUES (%s, %s)",
              ("price-history", msg.event_id))
        except UniqueViolation:
            return ack(msg)          # already processed — suppress, do not re-run
        tx.execute(
          "INSERT INTO price_history (wine_id, price_usd, observed_at) VALUES (%s,%s,%s)",
          (msg.wine_id, msg.new_price_usd, msg.occurred_at))
    ack(msg)

Brokers advertise exactly-once. What they generally provide is deduplication across one specific hop under specific conditions — producer-to-broker deduplication within a session or window, or transactional read-process-write confined to the broker's own storage. Those features are real and useful. They are also scoped to territory the broker controls, and your side effects usually are not on that territory. No broker on earth can prevent a duplicate email from a third-party mail API, because it cannot see the API, cannot roll it back, and does not know the send happened.

So the end-to-end guarantee is assembled, not purchased. At-least-once delivery from the broker, plus deduplicated processing at the consumer, plus a dedup key that survives redelivery, produces a system in which each real-world effect happens once. That is exactly-once as an illusion — engineered, verifiable, and honest about its parts.

Two primitives, one difference. A point-to-point queue delivers each message to exactly one of N competing consumers: add workers and you process the same stream faster, because each message goes to whoever is free. A pub/sub topic copies each message to every subscriber: add a subscriber and a new party learns about everything, without the producer changing at all.

Competing consumers scale one job. Fan-out informs independent parties. The consumer group is the construct that lets one topic serve both shapes — each group receives every message, and within a group the messages are split among members — which is why modern brokers blur the vocabulary. The underlying question never blurs: who needs to know? If the answer is “exactly one worker, whichever is free,” you want a queue. If it is “three teams with different reasons,” you want a topic, and each subscriber should have its own queue behind it so one slow consumer cannot stall the others.

Two messages travel through the wine-cellar pipeline and they are grammatically different in a way that encodes an architectural difference. wine.price_changed is an event: past tense, a statement of fact. It asserts that something happened and takes no position on what anyone should do about it. send_alert_email is a command: imperative, an instruction. It asserts that a specific action should be performed by a specific handler.

The distinction is not stylistic. It determines who owns the decision. When you publish an event, consumers decide what it means: alert-matching decides that a price crossing a threshold warrants an alert; price-history decides it warrants a row. Adding a fourth interpretation later requires no change to the producer. When you send a command, the sender has already decided — it has determined that an email should be sent, to whom, and why — and the receiver's job is execution.

Both are legitimate; the failure is confusing them. A producer that publishes email_requested to a topic “so we stay event-driven” has issued a command in an event's costume. The name says a fact occurred; the content dictates an action; the decision that an email is warranted lives in the producer while appearing to live in the subscriber. Now nobody owns the alerting policy: the producer cannot change it without breaking subscribers who assume it, and the subscribers cannot change it because they did not make it.

Once you have chosen an event, one question remains: how much does it carry? A thin event carries identifiers and a reference — enough for the consumer to go fetch what it needs. A fat event carries the state the consumer will need, so no callback is required.

Neither is free, and the cost is a coupling in both cases; the choice is which coupling you can afford. Thin events couple consumers to your availability: every consumer must call back to your API at processing time, so your outage becomes their outage and your rate limits become their throughput ceiling — and because the fetch happens later than the event, they may read a state that has since changed, which quietly re-introduces the ordering problems of module 6. Fat events couple consumers to your schema: every field you ship becomes a field somebody might depend on, and refactoring an internal name becomes a breaking change you discover in someone else's incident channel.

For wine.price_changed the wine cellar publishes fat, and marks exactly four fields as contract: event_id, wine_id, new_price_usd, and occurred_at. The rest — wine_name, merchant_id, and the display fields that will inevitably accrue — are documented as convenience data subject to change. A consumer that reads wine_name for an email subject line is making an informed bet; a consumer that keys its logic on an undocumented internal reference is not, and now knows it.

Everything so far has treated events as messages in transit and current state as the source of truth. Event sourcing inverts that: the ordered log of events is the source of truth, and current state is a projection derived by replaying it. The wine's price is not a column you update; it is what you get when you fold every wine.price_changed event for wn_4417 in order.

What it buys is real. Perfect audit: the log is not a summary of what happened, it is what happened, and “who changed this price on March 3, and to what?” becomes a query rather than an archaeology project. Replay: a projection with a bug can be rebuilt by re-deriving it from the log, and a new projection — a question you did not know you had in 2024 — can be answered retroactively over history you never thought to record because you recorded everything.

The costs are equally real: projection code to write and operate, projections that must be rebuilt when their logic changes, event schema evolution over years of immutable history, and a query model your team has not used. The heuristic is narrow on purpose. Adopt event sourcing where audit and replay are product requirements — regulatory, financial, or contractual — not where they are aesthetic preferences. Price history in the wine cellar qualifies, and is in fact event-sourced by nature: the price-history consumer's append-only table is a projection of the event log. User preferences do not qualify, and event-sourcing them buys a change history nobody will ever read at the cost of machinery everyone will maintain.

The email provider returns 503 Service Unavailable. The price payload arrives with new_price_usd missing. Both surface in your consumer as an exception, both increment the same error counter, and both will be retried identically by a naive handler. They could not be more different.

A transient failure is one where the same input may succeed later: a 503, a connection reset, a lock timeout, a rate limit. Patience is the correct response, because the world will change. A permanent failure is one where the same input will fail every time: a schema violation, a reference to a wine that does not exist, an amount that fails validation. Patience is worthless, because the world changing does not change the message.

So the first thing a consumer must do with an exception is classify it, and the classification should be explicit in the code rather than inferred from a retry count. In the wine-cellar consumers this is a small dispatch at the top of the error handler:

except ValidationError as e:         # permanent — the message can never parse
    dead_letter(msg, reason=str(e))
except (Timeout, ProviderUnavailable, RateLimited) as e:   # transient
    raise                            # let the broker redeliver after backoff
except Exception as e:               # unknown — treat as transient, but bounded
    log.exception("unclassified", event_id=msg.event_id)
    raise

The third branch matters. Unknown failures are treated as transient because the alternative — dead-lettering anything you did not anticipate — throws away recoverable work on the first unfamiliar exception. But they are bounded by the retry budget like everything else, so an unclassified failure that is in fact permanent still lands in the dead-letter queue rather than looping forever. The receive limit is the backstop for your incomplete classification, not a substitute for having one.

Having decided a failure deserves patience, the question is how much. Retrying immediately, repeatedly, against a dependency that just failed is how you convert someone else's brownout into their outage — and, if they are a shared dependency, into yours.

Exponential backoff spaces attempts by a growing interval — 1 s, 2 s, 4 s, 8 s, 16 s — so a recovering dependency gets progressively more room. That solves the intensity problem for one client. It does not solve the coordination problem for many, and the coordination problem is the one that takes systems down.

Here is the mechanism, with wine-cellar numbers. The email provider browns out for 90 seconds. During the brownout, 200 notification workers each fail a send. All 200 failed at nearly the same instant, so all 200 retry at t+1 s, all 200 retry at t+2 s, and so on. Each retry wave arrives as a synchronized spike against a provider that is trying to recover. The provider's capacity comes back at 40% and is immediately consumed by a 200-request burst, which fails, which schedules the next synchronized burst. The recovery collapses repeatedly and the outage outlives its cause — a self-inflicted denial of service, launched by your own fleet, in perfect lockstep.

Jitter is the fix, and it is one line: instead of waiting exactly base × 2^n, wait a random duration in [0, base × 2^n]. The waves flatten into a spread, the recovering provider sees a smooth ramp instead of a wall, and the fleet stops attacking in formation.

A poison message fails deterministically on every attempt. In the wine cellar it looks like this — the canonical event, malformed by a feed adapter bug:

{
  "event_id":      "evt_9f31",
  "type":          "wine.price_changed",
  "occurred_at":   "2026-03-14T09:12:04Z",
  "wine_id":       "wn_4417",
  "wine_name":     "Ch. Montrose 2016",
  "merchant_id":   "mch_082",
  "old_price_usd": 189.00,
  "currency":      "USD "
}

It is valid JSON, so nothing rejects it at publish time. new_price_usd is absent, and currency carries a trailing space that fails the ISO-4217 check. Every consumer that receives it raises the same validation error, forever. Without a bound, this message is immortal: it is received, it fails, its visibility timeout expires, it becomes available, and it is received again — consuming throughput, filling logs, and in an ordered partition potentially blocking everything behind it.

The bound is the retry budget, usually expressed as a max receive count. When a message has been received more times than the limit, the broker moves it to a dead-letter queue instead of making it available again. That single move converts an infinite loop into a finite, inspectable artifact: the message is out of the hot path, its body and metadata are preserved, and its existence is now a number you can alarm on.

A dead-letter queue with no alarm and no owner is not a safety net. It is a place where data goes to expire quietly, usually after 14 days, with better branding than /dev/null. The mechanism is only worth what its triage process is worth, so here is the full narrative for the malformed price event above.

Detect. The alarm is not on DLQ depth — depth can sit at zero for weeks and then jump to one, which is exactly the case you must catch. It is on age-of-oldest-message on the DLQ exceeding 15 minutes, paired with a low-threshold depth alarm at >= 1 for a five-minute window. Any dead letter is an event; a dead letter nobody has looked at in 15 minutes is a page.

Diagnose. The operator pulls the message without consuming it, and reads three things: the body, the failure reason recorded at dead-letter time, and the receive count. Here the body is missing new_price_usd and the reason is ValidationError: new_price_usd required; currency 'USD ' not ISO-4217. Receive count is 1 — it was classified permanent and dead-lettered immediately, which itself tells the operator this was never a provider blip. A query for sibling messages shows 340 more from mch_082 in the same 20-minute window, which converts an incident about one message into an incident about a feed adapter release.

Decide. Three options, and the decision belongs to the price-pipeline on-call, not to whoever noticed. Patch the producer and redrive — correct here, because the events represent real price changes users are owed. Discard with a record — correct when the messages are duplicates of data since superseded. Quarantine — correct when the decision needs a person who is not available at 3 a.m.

Redrive, after the fix. The order is not negotiable: the feed adapter is fixed and deployed first, and only then are the 341 messages moved back to the main queue. Redriving first means 341 messages take another lap and return to the DLQ, and if any of them had partially processed before failing, it means partial work repeated. That the redrive is safe at all is entirely due to the processed_messages ledger from m3 — the price-history consumer that already appended a row for a partially-processed event will suppress the second append on the redrive.

Total order means: if message A was published before message B, every consumer observes A before B. To provide that, some component must decide the single global sequence — and every message must pass through it. That component is a serialization point, and its throughput is the system's throughput. You may run a hundred consumers; you will still be limited to what one node can sequence.

Worse, that ceiling comes with a coupling. A single global order means a single stream, which means a slow message blocks every message behind it regardless of whether they are related. In the wine cellar, a globally ordered price stream would let one wine's slow-processing event delay alerts for every other wine in the catalogue — head-of-line blocking, applied to work that had no reason to wait.

Brokers therefore offer a different guarantee: order within a partition. A topic is divided into partitions, each an independent ordered log, each with its own consumer. Order is preserved inside each partition and explicitly not preserved across them. Total throughput scales with partition count, and the cost is that you must decide which messages need to share an order.

Nobody needs the Château Montrose price change ordered against the Barolo's. They are unrelated facts about unrelated objects, and observing them in either order harms nothing. What must be ordered is each wine's own history: if wn_4417 goes 189.00 → 154.00 → 161.00, a consumer that observes those out of order will leave the chart, and any derived current price, simply wrong.

That is what a partition key expresses. The key names the entity whose history must stay coherent; all messages sharing a key are routed to the same partition and therefore observed in publication order. For wine.price_changed the key is wine_id, and the guarantee purchased is precise: all events for one wine are observed in order; no ordering is guaranteed between different wines.

Choosing the key is therefore an act of naming the ordering you actually need. Ask what would be wrong if two messages arrived swapped. If the answer is “nothing,” they do not need to share a key. If the answer is “the derived state would be incorrect,” they do — and the key is whatever they have in common.

Partitioning distributes load only as evenly as the key distributes traffic. When one key carries a disproportionate share, its partition becomes a hot partition: a single lane with a growing backlog and a single consumer that cannot be scaled out, because a partition has exactly one reader within a consumer group.

The symptom is distinctive and easy to misread from aggregate dashboards. Total queue depth looks moderate. Average consumer lag looks fine. One consumer sits at 40 minutes of lag while the other three are idle. Any metric that averages across partitions hides this completely, which is why per-partition lag is the gauge that matters — a point module 8 returns to.

The remedies, in order of preference. Choose a better key. If merchant_id is skewed because one merchant is 45% of your volume, wine_id may distribute far better while still buying the order you actually need. Increase partition count. This helps only if the skew is a collision artifact rather than a genuinely dominant key; if one key is hot, more partitions leave it exactly as hot. Salt the key. Route wn_4417 to wn_4417#1 through wn_4417#4, spreading one entity across four partitions.

The wine cellar has one source of truth for a price — the wines row written at feed intake — and several derived copies: the price-history series, the search index, the analytics rollup, and each user's alert state. Every copy is updated by a consumer, and each consumer has its own lag.

“Eventually consistent” therefore is not one property of the system. It is a set of windows, one per consumer, each measurable and each different by orders of magnitude:

Two consequences follow. First, every consistency conversation should name a surface: “the search index is 2 seconds behind at p99” is a fact you can alarm on, while “the system is eventually consistent” is a shrug. Second, the windows are already instrumented — consumer lag from module 8 is the consistency window, expressed in the same seconds. You are not adding new observability; you are reading existing observability as a product property.

Users tolerate a two-second-stale search result and do not tolerate their own action vanishing. The lag that generates support tickets is almost always read-your-writes lag: someone creates an alert, the list page is served from a projection that has not caught up, the alert is not there, and the user concludes the product lost their data. Nothing is broken; the report is nonetheless correct as a description of the experience.

The fixes are targeted, and the targeting is the point. You do not need global strong consistency to solve this; you need the author to see their own write. Three patterns, in increasing cost:

Read the author's own writes from the source of truth. The alert list, when requested by the user who just created an alert, is served from the primary rather than the projection — for that user, for a bounded window after the write. Everyone else keeps the cheap read.

Render the enqueued action as an explicit pending state. When the work genuinely has not happened, say so: “Alert created — we'll check prices within a minute.” This is not a consolation prize. It is more informative than a checkmark, because it tells the user both what is true now and what to expect next.

Include the write in the response. The API that accepted the creation already knows the resulting object; returning it lets the client render the new state without any read at all.

The wine cellar's alert page states, verbatim: “Alert active. We check prices continuously; alerts arrive within 5 minutes of a price change.” That sentence is a promise with a number in it, and it is the same number the m8 worksheet derives alarm thresholds from. Consistency claims in the interface and alarm thresholds in the runbook should be the same figure — otherwise one of them is fiction.

Reads were the easy half. The hard half is a business operation spanning several services, each with its own database: reserve a bottle, charge the card, notify the buyer. You want all three or none. What you want is a transaction, and across service boundaries you cannot have one at acceptable cost.

The mechanism usually proposed is two-phase commit: a coordinator asks every participant to prepare, then tells everyone to commit. It works, and it couples the availability of every participant to the availability of every other, plus the coordinator. Participants hold locks across the whole protocol — including across a payment provider's 30-second timeout — so a slow third party becomes a bottle-inventory lock held for 30 seconds, blocking every other buyer of that wine. And if the coordinator dies after prepare, participants sit locked and undecided until it returns.

A saga replaces this with a sequence of local transactions, each committed immediately, each paired with a compensating action that semantically unwinds it if a later step fails. Nothing is held. Intermediate states are visible — a reservation exists briefly for a purchase that will not complete — and that visibility is the price of staying available.

The word “compensating” does real work and is routinely misread as “undoing.” A compensating action is a new forward action that produces a semantically acceptable outcome. It does not rewind the world, and designing sagas as if it did produces compensations that are fiction.

Sort every step into three categories before writing a line of code. Reversible: the compensation restores the prior state cleanly — release a reservation, and inventory is as it was. Compensable with residue: the compensation is adequate but leaves a trace — a refund reverses a charge, but the customer's bank shows a pending charge for a day, and both entries appear on the statement. Irreversible: no compensation exists — an email in someone's inbox cannot be unsent, and “delete the notification row” compensates your database rather than the world.

The design rule follows directly: sequence irreversible steps last. If notify runs after charge, a failed charge means the notification never ran and needs no compensation. If notify runs first, every downstream failure requires a correction email, which is a worse product experience and a permanent one.

A queue answers “when this happens.” A scheduled job answers “at this time.” The distinction sounds trivial and decides architectures.

Jobs suit periodic sweeps over state: reprice every cellar nightly, expire stale reservations, send the weekly digest, delete records past retention. The work is defined by a clock and a query, not by an event, and would still need doing if nothing had happened. Queues suit reactions: something occurred, and work follows from it.

Forcing either into the other's shape produces the ugliest systems in this guide. A job that polls a table for “things that changed” is a queue with worse latency, no delivery guarantee, and hand-rolled redelivery — sometimes a fine trade, as module 2's SKIP LOCKED quiz allowed, but a trade you must make knowingly. A queue used for periodic work means scheduling by publishing a message to yourself with a delay, which reinvents cron with fewer guarantees and no visibility into whether the schedule is still running.

The wine cellar's secondary example is a genuine job: the nightly cellar-valuation run at 02:00 UTC, which reprices every user's cellar against the latest prices and writes a valuation row per cellar per day. It is a sweep over all state, driven by a clock, and it would run identically whether or not any price changed that day.

Three wounds, all old, all still shipping.

Overlap. The valuation job normally runs 02:00–02:50. One night the catalogue grew and it runs 70 minutes; at 02:00 the next invocation starts while the previous is still writing. Two instances now sweep the same cellars and write the same valuations rows, double-counting anything they both touch. The fix is a distributed lock with a TTL — the second instance tries to acquire, finds it held, and follows a policy. The policy is the part teams omit: skip (log and exit, correct for idempotent nightly sweeps where tomorrow's run will catch up) or queue (wait and run after, correct when every run must happen). Choosing at 3 a.m. during an incident is choosing badly.

Missed runs. A deploy freeze eats the 02:00 slot; the job simply does not run. Cron has no memory and will not tell you. You need an explicit catch-up policy decided in advance — run it manually against a backfill window, or skip the day — plus a heartbeat alarm that fires when the job has not completed in over 26 hours. A job whose absence produces no signal is not scheduled; it is optional.

Timezones. Scheduling at 02:30 local time means that one night a year 02:30 does not exist and another night it happens twice — so the job either silently skips or runs twice, and both are DST doing exactly what it says on the tin. Schedule in UTC. If a job must align to local business hours, compute the offset explicitly and write down which side of the DST boundary you chose.

A queue is invisible by construction: no user is waiting on it, so nothing complains. Three gauges make it visible, and the reason to run all three is that each detects failures the others miss.

Queue depth — how many messages are waiting. Detects a consumer that has stopped or fallen behind arrival rate. Blind to a small number of messages that are stuck, and blind to skew when it is aggregated across partitions.

Consumer lag — how far the consumer trails the newest message, in messages or in time. Detects falling behind, and per-partition lag detects the hot partition from module 6 that aggregate metrics hide. Blind to a queue that is empty because the producer has stopped.

Age-of-oldest-message — how long the unluckiest message has been waiting. This is the gauge that catches what the others cannot: a single poison message cycling through receive-fail-redeliver keeps depth near zero and lag near zero while its age climbs without bound. Depth says one; lag says fine; age says forty minutes and rising.

Alarm on trends and ages, not only on absolute depth. Depth's correct threshold depends on drain time: depth ÷ consumer rate gives the minutes until the queue is clear, and that number — not the raw count — is what the user-facing promise constrains.

A trace that ends at publish makes your API look guilty for your consumer's sins. The support ticket says “the alerts feature is slow”; the trace shows a request that completed in 80 ms; the investigation stalls, because the four minutes the user actually experienced happened in a process the trace never entered.

The async-specific move is small: trace context rides inside the message. The producer injects the current trace and span identifiers into the message's attributes — not its body, since the payload is a business contract and this is transport metadata — and the consumer extracts them when it receives the message and starts its span with a link back to the producer's.

A link, not a parent-child edge, and the distinction is load-bearing. Async parent-child spans are actually permitted — a child may begin after its parent has ended — but a link is the better fit here for two reasons. First, the producer does not await the consumer: the producer span ended at publish, and the consumer span begins whenever it begins — 40 seconds later, or 40 minutes later after a redrive — so the link says “this work descends from that request” without pretending the request stayed open waiting for it. Second, it handles fan-out correctly: one publish, three consumer spans, three links, no fiction about a single call tree.

The closing sweep. Each entry: the name, the symptom you would actually observe, the corrective, and where it was taught. This is the page to reread before a design review.