Fast Enough

Start with an uncomfortable observation: modern CPUs execute on the order of a billion instructions per second per core, and almost no web request is limited by instructions. The request spends its life waiting — for a value to arrive from memory, for a page to come off an SSD, for a database on the next rack to answer, for an API in another region to reply. The dominant term in nearly every latency budget you will ever debug is distance: how far the answer had to travel to reach the question.

This is not a metaphor. Signal propagation, serialization, queueing at each hop, and the software layers that mediate every boundary all scale with how many boundaries you cross and how far apart they are. A function call and a database query look identical in your source code — price = lookup(wine) either way — and differ by five orders of magnitude in cost. Abstractions are designed to make that invisible. Performance work is the discipline of making it visible again.

The corollary is what makes the frame useful in a design review. When someone proposes a fix, ask what distance it shortens. A faster serializer shortens no distance. A bigger instance shortens no distance. Moving a lookup out of a loop removes forty round trips, and that is a distance answer — which is why it will beat both.

Five rungs cover essentially every cost you will meet in application work. Memorize them; practitioners who know these cold reason about designs in seconds that otherwise take an afternoon of benchmarking.

Three properties of this ladder do most of the analytical work. First, the gaps are multiplicative, not additive: they are large but uneven — commonly 100× to 1000×, though the SSD-to-same-datacenter step is only a few× — so mixing rungs in a single budget usually means the lowest rung present dominates. Second, the biggest single jump is the one marked in the figure — leaving the machine. Inside the box you are trading nanoseconds and microseconds; the moment a request crosses to another host you are in a different unit of measurement. Third, the bottom rung is bounded by physics, not engineering. A round trip from Virginia to Frankfurt is about 4,000 miles each way as the crow flies (real fiber routes are longer); light in fiber covers roughly 125 miles per millisecond, so 100 ms is not a vendor's inefficiency you can escalate to. No one is optimizing that away.

The numbers above are correct and, for most people, useless — human intuition does not distinguish a nanosecond from a microsecond, so both read as instant. The fix is a fixed multiplier. Multiply every duration by 10⁹, so that one nanosecond becomes one second, and the ladder becomes something you can feel in your body.

An L1 cache hit is one second — a beat. A memory read is 1.7 minutes, long enough to make coffee. An SSD read is a full day. A round trip to the database on the next rack is six days — a working week gone, to fetch one number. And a cross-region API call is three years. Not three years of work; three years of waiting, doing nothing else, for one answer.

Notice what the human scale makes obvious that the millisecond scale hides. In real numbers, 41 sequential same-datacenter trips is "20 milliseconds of network," which sounds like nothing and gets waved through code review. At human scale it is nearly a year of standing in line — and the fix (fetch all forty-one rows in one trip) collapses it to a week. That is a 40× reduction in the dominant term, available for one line of code, and it is invisible unless you are counting hops.

Two failure modes account for most wasted performance effort, and both are attractive precisely because they feel like decisive action.

The first is optimizing before measuring. An engineer forms a hypothesis from reading the code — this JSON serialization looks expensive — and rewrites it. The rewrite is real work, often clever work, and it changes the p99 by nothing at all, because serialization was 3% of the budget and the other 90% was sitting in Figure 1.2's loop. The wine-cellar team lived this: a two-week effort to replace the templating engine shipped, benchmarked beautifully in isolation, and moved the page's p99 from 4.2 s to 4.1 s. The work was not wrong; it was aimed at the wrong component, and no amount of quality in the execution can fix an aiming error.

The second is caching without an invalidation plan. A cache is the highest-leverage tool in this guide, which is exactly why it gets deployed reflexively — SETEX price:{id} 3600, latency drops, ticket closed. Six weeks later a customer disputes a valuation because the page showed a price that changed last Tuesday, and nobody can say why, because nobody wrote down what invalidates the entry or how stale it is allowed to be. The performance win was real. It was paid for with a correctness liability that was never priced.

The method that replaces both runs in five steps, and the remaining modules are its expansion: measure the tail (module 2) → locate the bottleneck (module 2) → shorten the distance (modules 3, 5, 6) → state the staleness you bought (module 4) → verify at the tail, with module 7 covering the case where distance is not the problem at all and capacity is. Every step is falsifiable. That is the whole point: performance stops being a matter of opinion the moment you commit to numbers before and after.

Two numbers get called "performance" and they measure different things. Throughput is how much work a system completes per unit time — requests per second, rows per minute. Latency is how long one unit of work takes from question to answer. They are the width and the length of the pipe, and a pipe can be very wide and very long at the same time.

The wine-cellar dashboard during the incident showed a healthy 340 requests per second with no errors, and the team spent two days believing the system was fine. It was fine, in the sense the dashboard meant: it was accepting and completing everything asked of it. It was also taking 4.2 seconds to answer a collector who wanted to know what their wine was worth. Throughput was never the complaint.

Latency distributions are not bell curves. They are right-skewed: a dense cluster of fast responses and a long thin tail stretching to the right, produced by retries, cache misses, garbage-collection pauses, contended locks, and unlucky queueing. The arithmetic mean of such a distribution sits well to the left of the pain — it is dragged toward the dense cluster where most requests live, and the tail is too thin to move it much.

The wine-cellar numbers make this concrete. Mean response time: 180 ms. p99: 4.2 s. Both are correct measurements of the same traffic. The mean was on the dashboard the team looked at every morning; the p99 was the reason support tickets used the word "broken."

Read the percentiles precisely, because the loose reading causes real errors. p99 is the value below which 99% of requests complete — it is not "the worst case" and it is not an average of the slow ones. One percent of requests are worse than p99, and some of them are far worse. When you fix a tail, you move a percentile; you do not eliminate a maximum.

The obvious objection to caring about p99 is arithmetic: it is one request in a hundred. Two amplifiers make that reasoning wrong.

The first is fan-out amplification. A modern page rarely makes one backend call. The value-my-cellar view makes about twenty — cellar, prices, valuation, user profile, feature flags, recommendations, and so on — and the page is not done until the slowest one returns. If each call independently has a 1% chance of landing in its own tail, the chance the page avoids all twenty is 0.99²⁰ ≈ 0.82, so roughly 18% of page loads wait on at least one tail outlier. A one-in-a-hundred backend event became a one-in-six user event, purely through composition.

The second amplifier is usage skew, and it is the one that turns a technical problem into a commercial one. Requests are not distributed evenly across users. The collector with 40 bottles loads the valuation page occasionally; the collector with 900 bottles loads it several times a day, triggers the heaviest queries, and is on your most expensive plan. If a user issues 200 requests in a session, their chance of meeting at least one p99 outlier is 1 − 0.99²⁰⁰ ≈ 87%. Your tail is not sampled uniformly by your user base — it is sampled disproportionately by the users you can least afford to lose.

Once you know the page is slow at the tail, the next question is where, and the answer must come from a measurement, not from reading the code. Intuition nominates suspects; a profile convicts. The distinction matters because intuition is systematically biased toward code that looks expensive — nested loops, big string operations, regular expressions — while actual budgets are usually consumed by things that look like nothing at all, like a property access that lazily triggers a query.

Two forms of evidence cover most application work. A profile attributes time to code paths, telling you which functions or spans own the budget. A distributed trace or query log attributes time to hops, which per module 1 is usually where it went. Use both; the query log frequently settles the case on its own by revealing a statement count nobody expected.

Then apply the constraint that saves you from wasted sprints. Amdahl intuition: speeding up a component buys you at most that component's share of total time, no matter how good the optimization is. A component that is 5% of the budget caps out at a 5% improvement even if you make it infinitely fast. So the profile's real job is not to find something slow — plenty of things are slow — but to find the component that is the budget.

The N+1 query is the most common serious performance bug in application code, and it is worth studying as a specimen because its mechanism generalizes. The ORM loads the cellar in one query. Then the rendering loop touches bottle.price, which the ORM has helpfully declared a lazily loaded relation, and each touch issues its own round trip. One query for the list, N for the items: 41 trips for 40 bottles.

What makes it archetypal is that the code is good. There is no nested loop, no obvious waste, nothing a reviewer would flag. The cost is not in the code that is written; it is in the boundary the code silently crosses, once per row. Module 1's lesson lands here: the hierarchy's costs hide behind abstractions designed to make a six-day round trip look like a property access.

The fix is a join or a batch fetch — IN (...), or the ORM's eager-loading directive — and the deeper practice is to make query counts visible: log statements per request in development, and alert when a request issues more than a threshold. Guide Nº 06 covers the SQL mechanics and the ORM directives; what belongs here is the recognition skill. The signature is unmistakable once you have seen it: many fast queries rather than one slow one. A team that has not learned that signature reliably misdiagnoses N+1 as "the database is slow" and buys a bigger database, which changes nothing, because each individual query was already fast.

A cache is a copy of an answer stored nearer to the question than the answer's source of truth. That is the entire definition, and it is worth being pedantic about each part. It is a copy — the source still exists and can still change, which is where all the trouble in module 4 comes from. It stores an answer, not data as such: a rendered page, a query result, a computed valuation, a fetched price. And nearer means nearer on module 1's hierarchy — fewer rungs down, fewer boundaries crossed.

Once you hold that definition, you start seeing it everywhere, because it is the same idea implemented at every scale of the machine. The CPU's L1 cache keeps recently used memory a nanosecond away instead of a hundred. The operating system's page cache keeps file contents in RAM instead of on the SSD. A database's buffer pool keeps hot pages in memory instead of re-reading them. An application cache keeps a computed result in the process instead of recomputing it. A CDN keeps a response in a city near your user instead of a continent away. Different vendors, different words, one idea repeated across eight orders of magnitude.

The dominant pattern is cache-aside, and it is worth walking precisely because the miss path contains a fact people forget. Check the cache. On a hit, return the cached answer and you are done. On a miss, go to the source, get the answer, write it into the cache, and return it.

Look at what a miss cost you. You paid the cache lookup and the full source fetch and the cache write. A miss is strictly more expensive than not having a cache at all. It is usually only slightly more — a millisecond against eighty — but the sign matters: a cache is not free insurance that sometimes pays off. It is a bet that most lookups will hit, and if they do not, you have added latency, memory, an operational dependency, and a correctness liability in exchange for nothing.

One formula prices a cache. With hit rate h, hit cost t_hit, and miss cost t_miss, the effective latency is h·t_hit + (1−h)·t_miss.

Run it on the wine-cellar price feed, where a cached price is 1 ms away and the source is 80 ms. At h = 0.50: 0.5(1) + 0.5(80) = 40.5 ms. At h = 0.90: 0.9(1) + 0.1(80) = 8.9 ms. At h = 0.99: 0.99(1) + 0.01(80) = 1.8 ms.

Now notice the shape those numbers make, because it defeats most people's intuition. Going from 50% to 90% — forty percentage points — saves 31.6 ms. Going from 90% to 99% — nine percentage points, less than a quarter as much movement — saves 7.1 ms, and more importantly cuts what remains by a factor of five. The reason is that effective latency at high hit rates is almost entirely made of misses, so what matters is not the hit rate but the miss rate, and miss rate is what moves multiplicatively. 90% → 95% halves the misses. 90% → 99% removes nine-tenths of them. 99% → 99.9% removes nine-tenths again.

Two practical consequences. First, when you estimate a cache's value, estimate the reuse rate of the data — how many times an answer is read between changes — because that is what sets the achievable hit rate. The wine-cellar price for a given wine is read roughly 200 times between feed updates, which supports a hit rate well above 99%. Second, resist reporting hit rate as a single global number; it averages across keys with wildly different reuse and hides the ones dragging the tail down.

Everything above is the case for caching, and it is strong. Now the honest accounting, because a cache is not free and two entry mistakes are so common they deserve names.

The first is the cargo-cult cache: an in-memory store added because caching is what serious systems have, with no measurement of reuse beforehand. The wine-cellar team's first instinct in the incident was exactly this — they put Redis in front of the cellar-contents query, deployed it, and measured a 12% hit rate. The reason is obvious in hindsight: cellar contents are per-user and change whenever the collector adds a bottle, so there was almost no reuse to capture. Effective latency went from 45 ms to 0.12(1) + 0.88(46) ≈ 40.6 ms — a 10% improvement bought with a new production dependency, a new failure mode, and a new source of stale data. Not a catastrophe. Just a bad trade, made without the arithmetic that would have predicted it.

The second is indiscriminate caching: caching everything, on the theory that more caching is more speed. It fails on two fronts simultaneously. Memory is finite, so caching low-value entries evicts high-value ones and your hit rate on the keys that mattered falls. And every cached item is one more copy that can be wrong, so correctness surface grows linearly with the number of things you cache while the latency benefit concentrates in a few hot keys. The rule of thumb worth internalizing: cache the few things that are read many times, not the many things that are read once.

What you will notice about both anti-patterns is that neither is fixed by knowing more about caching. Both are fixed by module 2: measure the reuse, price the fix, then act. A cache is a fix like any other, and it earns its place with a number.

Before the cache, there was one answer and it was current by construction. After the cache, there are two, and the copy diverges from the source the instant the source changes. Staleness — the gap between the cached copy and current truth — is not a risk that might materialize. It is a schedule. Given a cache and a source that changes, staleness will occur; your design decides only how much and for how long.

This reframing is the whole module. "Is the cache correct?" is a question with no useful answer. "What is this cache's worst-case staleness, and is that acceptable for this data?" is answerable, testable, and reviewable. A price cache with a one-hour bound is fine for a browsing page and unacceptable for a checkout total. Same mechanism, same code, different answer, because the tolerance belongs to the data and the use, not to the cache.

A TTL (time-to-live) attaches an expiry to a cache entry: after N seconds, the entry is discarded and the next read goes to the source. Its great virtue is that it requires no cooperation from anything else in your system. Write paths do not need to know the cache exists. Bulk imports, admin consoles, database migrations, and the script someone runs manually on a Tuesday all work correctly, because the entry expires on its own schedule regardless of how the source changed.

Its great vice is that the staleness bound is exactly the TTL and not one second less. A 300-second TTL means that for up to five minutes after a change, every reader sees the old answer, and there is nothing you can do about it from the write side. TTL does not detect change; it forgets on a timer.

Choose N from the data's tolerance, not from habit. The value 300 appears in an enormous amount of production code for no better reason than that it appeared in a tutorial. Ask instead: how fast does this data actually move, and how much staleness does the reader's decision tolerate? Wine market prices update once daily, so a one-hour TTL is generous rather than risky. Feature flags need to take effect in seconds, so their TTL is measured in seconds — or they use explicit invalidation. An exchange rate on a payment page might tolerate 30 seconds. Each of those is a defensible number with a sentence of reasoning behind it, which is more than "300" ever has.

And remember module 3's arithmetic prices the choice. Shortening a TTL raises the miss rate, which raises effective latency and load on the source. If a key is read 200 times an hour and you cut its TTL from one hour to one minute, you have gone from one miss per hour to sixty, multiplying source load by 60× for a freshness improvement that may buy nothing the reader can perceive. TTL selection is a latency/freshness trade with real numbers on both sides, not a safety dial you turn down when nervous.

Explicit invalidation deletes or updates the cached copy at the moment the source changes. Its staleness bound is approximately zero — bounded only by propagation delay — which makes it the right mechanism whenever freshness genuinely matters.

It buys that precision with an obligation that is easy to underestimate: every path that can modify the source must fire the invalidation. Not most paths. Every path. The API handler you wrote does. The nightly bulk import someone added last quarter does not. The admin console's manual override does not. The data-fix script run during an incident certainly does not. And the failure is silent — no error, no alert, just one entry serving an answer that stopped being true, indefinitely, until something else happens to displace it.

This is the origin of "stale data no one can explain," a category of bug notable for how much time it consumes. Nobody can explain it because the invalidation code is present and correct and is simply never reached by the write path that actually changed the data.

The robust design is not to choose. It is belt and suspenders: explicit invalidation for freshness, plus a generous TTL as a backstop. The explicit path handles the normal case in milliseconds; the TTL bounds the damage of every write path you forgot, every deploy that dropped an event, every network partition that ate a message. Your worst-case staleness becomes the TTL, your typical staleness becomes near-zero, and the failure mode degrades from "wrong forever" to "wrong for at most an hour." That is a very large difference in an incident review.

Caching does not only mitigate failure modes; it introduces one. The cache stampede — also called a dogpile or thundering herd — happens when a hot entry expires while under concurrent load. Every request that arrives in the window between the expiry and the first refill finds a miss, and every one of them independently goes to the source. The cache was absorbing that traffic a millisecond ago. Now the source receives all of it at once.

Three mitigations, and mature systems use all three because they address different parts of the mechanism.

Jittered TTLs. Set the expiry to N ± a random spread — say 600 s ± 60 s — instead of exactly N. This does nothing for a single hot key, but it is essential at scale: when a thousand keys are populated together during a deploy or a cache flush, an identical TTL makes them all expire in the same second, producing a synchronized stampede across the entire key space. Jitter spreads that over a minute.

Single-flight. When a key misses, the first request acquires a per-key lock and recomputes; concurrent requests for the same key wait for that result instead of starting their own. Source load on a miss becomes exactly one query regardless of concurrency. This is the direct fix for the single-hot-key case: 800 requests, one recomputation.

Stale-while-revalidate. Keep the expired value and serve it while a background refresh runs. Users never pay the miss at all — the entry becomes stale-but-served for the refresh duration, then updates. It is the best user experience of the three and it explicitly extends the staleness bound to include the refresh window, which is a trade you must make deliberately and state.

Consider a single GET for a cellar summary and follow it down. The browser may answer it from its own HTTP cache with no network at all — zero on the hierarchy, the fastest possible outcome because the request never happens. Failing that, it reaches a CDN edge node in the user's city: one short network hop instead of a continental one. Failing that, it arrives at your application, which may answer from an in-process cache — a memory read. Failing that, the application asks a shared in-memory store: one same-datacenter round trip. Failing that, the database, which may still answer from its buffer pool or a precomputed view rather than scanning. And failing all of it, the full cost is paid: disk, computation, and the whole distance back.

Each layer is a rung, each hit short-circuits everything below it, and a full miss pays the entire depth. That structure is why placement is a real decision rather than a detail — moving a cache one layer up can save more than any amount of tuning within a layer.

The two outermost layers are the highest-leverage and the most dangerous, for the same reason: they are furthest from your code and closest to your users.

The browser cache is caching infrastructure you already ship in every response, whether you have thought about it or not. Cache-Control instructs every cache between your server and your user what may be stored, by whom, and for how long. A CDN then multiplies that decision across users: one origin fetch serves thousands of readers from an edge node near each of them.

Two properties define working at these layers. First, invalidation is blunt. You cannot reach into a user's browser and delete an entry; once you have served max-age=86400, that answer is out of your control for a day. CDNs offer purge APIs, but purges are eventually consistent across a global fleet and are an operational action rather than a code path. The technique that actually works is invalidation by renaming: give the content a URL derived from its contents (app.7f3c9d.js), cache it effectively forever, and ship a new URL when the content changes. There is nothing to invalidate because the old answer is still correct for the old URL.

Second, the blast radius is maximal. A wrong entry in a shared edge cache is served to every user who requests that URL. This is where a performance decision becomes a security decision: Cache-Control: public on an authenticated page tells the CDN it may store that response and serve it to anyone requesting the same URL — which means user A's cellar contents can be served to user B. The correct discipline is that anything varying by identity is marked private (browser may store it, shared caches may not), or is cached with the identity in the key, or is not cached at those layers at all.

Inside your own process, two options differ in a way that matters more than their similar-looking APIs suggest.

In-process memoization — a dictionary in memory, an LRU decorator — is the fastest cache that exists, because a hit is a memory read on module 1's second rung. It has no network cost, no serialization, no external dependency. Its defining property is that it is per replica: if you run eight application instances, you have eight independent caches that know nothing about each other. Invalidating means invalidating eight copies, which usually means you cannot invalidate at all and must rely on TTL.

That incoherence is fine for some data and user-visible for others. Configuration, feature-flag definitions, compiled templates, currency-code tables, and anything else that changes rarely and identically for everyone are ideal: an inconsistency window of seconds across replicas harms nothing. But cache a user-visible value in-process and the user's requests, load-balanced across replicas, will hit different copies with different ages — producing flapping: refresh the page and the total goes up, refresh again and it goes back down. This is one of the most confusing bug reports a support team can receive, because it is not reproducible from any single instance.

The shared in-memory store — Redis, Memcached, or equivalent — costs one same-datacenter round trip (~1 ms rather than ~0.1 ms) and buys one shared truth. All replicas see the same entry, invalidation is a single delete, and the cache survives a deploy that restarts every application process. For anything user-visible, that coherence is worth the extra hop, and the arithmetic from module 3 says so plainly: against an 80 ms source, the difference between a 0.1 ms and a 1 ms hit is invisible.

The bottom layer caches too, in two quite different ways worth separating.

The buffer pool caches pages of data in RAM so repeated reads do not touch the SSD. It is transparent, automatic, and requires nothing from you except enough memory. It also, per module 3's fourth quiz, saves the database's I/O and not your distance to the database — a buffer-pool hit still costs you a full round trip plus connection acquisition, parsing, planning, and serialization.

A materialized view is different in kind. It stores a query's answer as a table, so a query that scans a million price-history rows becomes a lookup of one precomputed row. That is unambiguously a cache by module 3's definition — a copy of an answer kept closer than its source — and it is therefore subject to everything module 4 taught. It has a refresh rule, which is its invalidation policy. It has a worst-case staleness, which is the interval between refreshes. It can serve wrong answers if its refresh is missed or fails silently.

The distinction that makes it valuable: this is the layer that fixes expensive computation, whereas the layers above fix distance. The wine-cellar valuation aggregation was 1.8 s of genuine database work — scanning price history, joining, summing — and no amount of caching further up would have made that computation cheap for the first requester of every distinct cellar. Precomputing it collapses the work itself.

The running example, assembled. Recall the diagnosis from module 2: p99 of 4.2 s, composed of ~2.1 s of N+1 price lookups and ~1.8 s of valuation aggregation. Three interventions, each at the layer that matches what it fixes, each with its invalidation rule stated at the moment it is introduced.

Fix 1 — batch the query. Before any cache, eliminate the N+1: one IN (...) fetch instead of 40 sequential round trips. This is not caching at all and it comes first because it removes work rather than relocating it. 2,100 ms → 55 ms.

Fix 2 — price feed to the shared in-memory store. Prices are shared across users with roughly 200:1 reuse, making them the ideal caching candidate from module 3's arithmetic. Invalidation rule: the price-feed ingest job deletes each changed price:{wine_id} at commit; a 1-hour TTL backstops any missed delete; single-flight protects against stampede on hot wines. Worst-case staleness 1 hour, surfaced as "prices as of 09:00 UTC." 55 ms → 4 ms.

Fix 3 — aggregation to a materialized view. The 1.8 s is computation, so it moves to the layer that precomputes. Invalidation rule: refreshed when the price feed lands and on any cellar mutation for that cellar; worst-case staleness equals the gap to the next refresh, which the same as-of timestamp discloses. 1,800 ms → 12 ms.

Fix 4 — public summaries to the CDN. Shared cellar views are identical for every reader, so they belong at the edge. Invalidation rule: purge on cellar edit; s-maxage=600 with ±60 s jitter to avoid synchronized expiry; authenticated pages marked private so no personalized response is ever eligible for the shared cache. Worst-case staleness 10 minutes for public views, disclosed on the page.

Five questions characterize a workload well enough to choose a store. Answer all five before any product name enters the conversation.

Read/write ratio. Is this read 1,000 times per write, or written continuously and read in batches? Read-heavy workloads have options (replicas, caches, denormalization) that write-heavy ones do not.

Lookup shape. Point lookup by exact key? Range scan over an ordered field? Multi-table join? Full analytical scan with aggregation? This is the question that most directly maps to a store family, and it is the one people most often skip.

Consistency need. Must a read reflect the write that just happened, or is a few seconds of lag invisible? Must multiple changes commit atomically, or is each independent?

Lifetime and durability. Does this data survive a process restart? A datacenter loss? Is it derivable from something else (a cache, an index, an embedding) or is it the only copy of a fact?

Size and growth. Does it fit in memory today? In three years? Does it grow with users, with time, or with both?

Each non-relational family buys its speed by shedding a guarantee. Name the guarantee you are shedding, because it is a price, not a footnote.

Key-value and in-memory stores (Redis, DynamoDB, Memcached) are optimized for point lookup by exact key, and they are exceptionally good at it — sub-millisecond, at very high throughput, scaling horizontally because keys partition trivially. What they shed: ad-hoc query. You can retrieve session:abc123 instantly, and you cannot ask "which sessions were created in the last hour by users in Germany" at all without maintaining a second structure yourself. They fit sessions, caches, counters, rate-limit buckets, feature flags, and job state — data whose access is "give me this exact thing," typically ephemeral or derivable.

Document stores (MongoDB, DynamoDB's document mode, Postgres's own jsonb) fit workloads where a nested aggregate is the unit of work: you read the whole document, you write the whole document, and nothing else needs to join into its middle. A tasting note with nested scores and tags is a plausible document. What they shed: cross-entity integrity. Without foreign keys and joins, referential correctness moves into application code, where it is enforced by discipline rather than by the engine — and discipline degrades under deadline in a way constraints do not.

Analytical stores and warehouses (covered in Guide Nº 20) are built for scans over large volumes with aggregation, typically column-oriented so a query touching three columns of a hundred reads three columns' worth of data. What they shed: fast single-row writes and point lookups. They are the wrong home for transactional state and the right home for "valuation trends across all cellars over 24 months."

The lowest-friction way to scale reads is a read replica: a copy of the primary database that receives its changes asynchronously and serves read-only queries. No schema change, no new query language, no shed guarantees — just more read capacity. It is genuinely the right move for read-heavy workloads, and it has one sharp edge that every team meets.

Asynchronous replication means the replica is always slightly in the past. Replication lag is usually milliseconds, occasionally seconds, and under load or during a long-running transaction on the primary it can be much worse. Most reads do not care. One category cares enormously.

Read-your-writes is the guarantee that a user who just made a change sees that change. It is not a technicality; it is the most basic expectation anyone has of software. The collector adds a 2016 Margaux to their cellar, the write commits to the primary, the browser redirects to the cellar list, that read is load-balanced to a replica that has not yet received the change, and the bottle is not there. The user adds it again. Now there are two.

The fix is routing, not replication tuning. Reads that must see the user's own write go to the primary: either by marking those code paths explicitly, or by session pinning (a user who wrote within the last N seconds reads from the primary), or by carrying a replication position forward and waiting for a replica that has caught up to it. All three are correct; all three are choices you make deliberately per flow.

What is not a fix is making replication faster. That shrinks the window in which the bug occurs, converting a reproducible failure into an intermittent one — which is strictly worse for diagnosis and no better for the user who lands in it. Any "fix" that changes the probability of a race without eliminating it deserves that suspicion generally.

Five workload classes cover most of what an application does, and each maps to a store family. The map is not a rule — it is a starting position that a stated access pattern can override.

Placing the wine-cellar app on the map produces a deliberately unexciting result. Sessions → in-memory store: point lookup by token, ephemeral, no query needs. Cellar records and users → relational, and they stay there: joins across owners, wines, and purchase records, constraints that must hold, and a write volume nowhere near a single instance's ceiling. Price history → relational, with the materialized view from module 5 handling the aggregation. Analytics on valuation trends → warehouse (Guide Nº 20), because scanning 24 months of history for reporting is exactly the scan pattern that does not belong on the transactional primary. Cached prices → in-memory store, per module 5.

Four of five workloads either stay in Postgres or use it plus a cache. That is the normal outcome of doing this exercise honestly, and it is worth stating plainly: the map's usual verdict is that your relational database is fine and one or two specific patterns want somewhere else to live.

Two ways to add capacity, with different limits.

Vertical scaling — a bigger machine — raises the ceiling without changing anything about your architecture. It is genuinely underrated: modern instances are enormous, an afternoon of vertical scaling frequently buys a year of headroom, and it introduces no new failure modes. Its limits are that machines have a maximum size, the price curve steepens badly at the top, and a single machine remains a single failure domain.

Horizontal scaling — more machines — multiplies capacity cheaply for stateless work and adds redundancy. Its limit is the one that matters in practice: your stateless tier can grow indefinitely and every request still converges on the same primary database, the same in-memory store, the same third-party API. Doubling app servers when the constraint is downstream buys nothing, which is module 2's Amdahl argument applied to capacity rather than to time.

And one property is easy to miss: horizontal scaling adds distance. More instances mean more coordination, more network hops between components that used to share a process, cache incoherence across replicas (module 5), and load-balancer indirection. It buys throughput, not per-request speed. A request may well be marginally slower on a horizontally scaled system than it was on one big box — which is a perfectly good trade when you need the capacity and a poor one when you were trying to fix latency.

Here is the fact that surprises people, and it is worth internalizing as a shape rather than a formula. As a resource's utilization rises, waiting time does not rise proportionally. It rises gently through the middle of the range and then goes nearly vertical as utilization approaches capacity.

The intuition, without any mathematics: a server is only idle-and-waiting when a request arrives at a moment nothing else is being served. As utilization climbs, that coincidence gets rarer, so more arrivals find the server busy and must wait behind work in progress. Near saturation, arrivals almost always find the server busy, and each new arrival waits behind a queue that itself has not had time to drain. Small increases in arrival rate then produce very large increases in wait, because the queue's drain rate has no margin left. Traffic is bursty — it arrives in clumps, not evenly — which is why the effect bites well before the theoretical 100%.

Three consequences follow, and they are the reason this shape is worth knowing.

Headroom is a design number. Running at 50–70% is not inefficiency; it is the purchase of the flat part of the curve. A finance-driven push to "use what we pay for" by running at 90% is a request to operate past the knee, where a 10% traffic increase does not cost 10% more latency — it can cost several hundred percent.

Autoscaling does not repeal the curve; it races it. Autoscaling reacts on a timescale of tens of seconds to minutes — detect, provision, boot, warm, join the pool. The cliff acts in seconds. Autoscaling is real capacity insurance for sustained load and no protection at all against a burst, which is why it must be paired with the backpressure of the next section rather than treated as a substitute for headroom.

Utilization must be measured on the scarcest resource, which is usually not CPU. It may be database connections, a thread pool, file descriptors, an in-memory store's single-threaded command loop, or a third-party rate limit. The wine-cellar stampede saturated database connections while application CPU sat under 30% — the dashboard everyone was watching showed a healthy system throughout.

A connection pool is the most common capacity control in application code, and it is routinely misunderstood as a performance optimization. It is not. It is a bound — a deliberate queue placed in front of a resource that degrades catastrophically under unbounded concurrency.

A database can serve some number of concurrent queries well. Past that, each additional concurrent query does not get its own slice of the machine; it adds contention for locks, buffer pool, and CPU scheduling, so all queries slow down together and total throughput falls. The pool prevents you from getting there: it caps concurrent connections and makes additional requests wait in your application, where waiting is cheap and observable, rather than inside the database, where it is expensive and opaque.

Which is why raising a pool from 50 to 500 because "requests are waiting for connections" is exactly backwards. The requests were waiting; now they proceed, all five hundred of them, into a database that could serve perhaps eighty concurrently. The queue did not disappear — you relocated it from a place you control to a place you do not, and you removed the protection that was working. Size the pool near what the downstream can genuinely serve, and let the wait happen in front of the pool where you can see it, time it, and shed it.

Which brings us to the humane response to genuine overload. Backpressure is refusing or bounding new work when you are full: bounded queues that reject when they fill, load shedding with an honest 429 Too Many Requests and a Retry-After, timeouts that free resources rather than letting a request wait forever for a client that has already left.

Module 1's top two rungs have consequences inside a single process that are worth recognizing even though you will rarely tune them directly.

A CPU does not fetch one byte from memory; it fetches a cache line — typically 64 bytes — and it aggressively predicts sequential access, prefetching ahead of what you have asked for. So iterating a compact array of numbers is enormously faster than following pointers between objects scattered across the heap: the first pattern gets many useful values per fetch plus successful prediction, the second gets one value per fetch and defeats the prefetcher. The same loop, over the same count of the same values, can differ by an order of magnitude based purely on memory locality.

You are unlikely to need this for a typical web request, where a 100 ns memory read is invisible next to a 500 µs round trip. It matters in the places where you process many items in a tight loop: the valuation aggregation over hundreds of thousands of price rows, an embedding similarity scan, an image or CSV transform, anything sorting or grouping large collections. When a profile shows a loop taking far longer than its arithmetic suggests, locality is a plausible suspect and "process a compact array rather than a graph of objects" is usually the entire remedy.

The last inherited cost is the most distinctive to diagnose. A garbage collector reclaims memory your program is no longer using, and depending on the runtime and collector, it may need to stop the world — pause all application threads — to do part of that work. Every request in flight at that moment stops. None of them did anything wrong.

The signature is what makes it identifiable, and it is worth learning by sight because the alternative is weeks of blaming the database. Plot p99 latency and request rate on the same time axis. A capacity problem shows latency rising with traffic. A GC pause shows sharp latency spikes on flat traffic, often on a rhythm — every thirty to sixty seconds, or after a heavy allocation burst — with a consistent height corresponding to the collector's pause duration. The correlation with traffic is the tell: there is no request behind the spike.

The responses are runtime-level, not application-level: reduce allocation rate in hot paths (the largest lever, and the one most under your control), tune heap sizing and generation ratios, or move to a low-pause collector designed to trade throughput for predictable pause times. Which applies depends on the runtime, and that is Guide Nº 25's territory.

What matters here is the negative result: no cache fixes a GC pause. The pause stops the thread that would read the cache. Neither does a bigger database, more replicas, or a CDN. This is the clearest case in the course of a latency problem that is not a distance problem, and every tool from modules 3 through 6 is simply inapplicable — which is exactly why you diagnose before you reach for a lever.

Add a rung to module 1's ladder, below everything else on it. A model call takes on the order of 1 to 10 seconds. On the ×10⁹ human scale that is somewhere between thirty and three hundred years — a cost so far outside the rest of the hierarchy that it changes what kind of thing a request is. You do not optimize around a model call the way you optimize around a database query; you architect around it.

It also has internal structure that a single latency number hides, and the structure is what you act on.

Time-to-first-token (TTFT) is the delay before any output appears. It is composed of queueing at the provider, plus prefill — the model processing your entire input prompt before it can produce anything. TTFT therefore scales with input size and with how busy the provider is. It is what you attack by shortening prompts, trimming retrieved context, using provider-side prompt caching, or choosing a less congested deployment.

Tokens per second is the generation rate after the first token. Total time ≈ TTFT + (output tokens ÷ generation rate). It scales with output length and with model size, so you attack it by generating less — asking for a shorter answer, returning structured fields instead of prose, splitting one long generation into parallel shorter ones — or by using a smaller model.

Streaming sends tokens as they are generated rather than withholding the response until it is complete. Be precise about what it does: it changes nothing about total generation time. An 8-second generation still takes 8 seconds. What it changes is when the user sees the first evidence that something is happening — from 8 seconds to perhaps 800 milliseconds.

That is not a trick, and it is not merely psychological. A user watching text appear can begin reading, begin evaluating, and decide to stop early if the answer is going the wrong way. Perceived latency is a real quantity that determines whether a feature feels usable, and streaming is the highest-leverage tool for it — comparable in impact to a genuine 10× speedup, at a fraction of the cost.

And now the exception, which is where streaming stops being free. Streaming shows the user an incomplete answer, and an incomplete answer can be wrong in a way the complete answer is not. "The transaction appears compliant with the applicable reporting threshold—" is a sentence a reader can act on. The tokens that follow may be "—however, the counterparty's jurisdiction triggers a separate obligation that has not been satisfied."

The rule: stream when partial output is informative but not actionable; withhold when partial output is actionable and reversible. A tasting note, a draft email, an explanation, a summary — stream them. A compliance determination, a computed total, an approve/deny, a risk score — withhold, or restructure so the conclusion arrives atomically at the end while the reasoning streams before it. That restructuring is usually the best of both: the user sees progress and cannot act on a verdict that has not been rendered.

Model calls are the most expensive thing in a modern request budget, in both latency and money, which makes them the highest-value caching target in the course. Three distinct caches, with three distinct correctness profiles.

Embeddings are the easy case and the one teams most often miss. An embedding is deterministic: the same input through the same model version yields the same vector. So it can be cached effectively forever, keyed on a hash of the content plus the model identifier. There is no staleness risk while those two are unchanged, and re-embedding an unchanged corpus on every deploy is pure waste — it is common, and it is usually one of the largest available savings in a retrieval pipeline.

Exact-match response caching keys on the full normalized prompt. It is safe — identical input, identical output — and its yield depends entirely on how repetitive your prompts are. For a free-text assistant, near zero. For a feature generating tasting notes from a fixed template with a small set of wine attributes, substantial. Module 3's rule holds unchanged: measure reuse before building it.

Semantic caching serves a cached answer to a question that is merely similar by embedding distance. It converts a near-zero hit rate into a meaningful one, and it does so by deliberately trading correctness for hit rate. That is a legitimate trade on tolerant surfaces and a serious hazard on precise ones: "what is my 2016 Margaux worth" and "what is my 2015 Margaux worth" are close in embedding space and have different correct answers. If you deploy semantic caching, name the trade explicitly, set the similarity threshold conservatively, and keep it away from anything a user acts on.

Provider-side prompt caching is a fourth mechanism worth knowing: providers can cache the processed form of a repeated prompt prefix, cutting both TTFT and input cost on subsequent calls. It rewards structuring prompts with stable content first — system instructions, retrieved documents, few-shot examples — and the variable user input last.

Every AI feature occupies a point among three quantities: model quality (how good the output is), latency (how long the user waits), and cost (what each call spends). You can trade between them. You cannot escape them, and the corners move together — a larger model is generally better, slower, and more expensive at once.

What distinguishes teams here is not where they land but whether they chose. The default behavior is to pick the strongest model during prototyping, ship it, and discover the position months later via a cost dashboard or a complaint. The discipline is to name the corner you are sacrificing before you build, and to write down what would change the decision.

Everything in this course reduces to one loop. It is deliberately small enough to run from memory in a design review.

1 · Measure the tail. p99, not the mean, over a stated window, segmented by whatever dimension the cost scales with. Count the fan-out. (Module 2.)

2 · Locate the bottleneck. Profile and trace; rank components by share of the budget; compute each candidate fix's Amdahl ceiling before doing any work. (Module 2.)

3 · Shorten the distance. Remove work first if you can — batch the N+1, add the index. Then move the answer closer: the right cache at the right layer, or the right store for the access pattern. (Modules 1, 3, 5, 6.)

4 · State the staleness. Every cache you introduced gets an invalidation trigger, a TTL backstop, a worst-case staleness number, and — where a user relies on the value — a user-facing statement of it. (Module 4.)

5 · Mind the cliff. Check that the fix did not just move the constraint. Confirm headroom on the scarcest resource, bound the queues, and verify backpressure exists. (Module 7.)

6 · Verify at the tail. The same measurement as step 1, after. If p99 did not move, the fix did not work, whatever the microbenchmark says. (Module 2.)

And the index. Every anti-pattern this course named, with the symptom that reveals it in the wild and the corrective.