What Green Means · Nº 26
A field guide to software testing
Somewhere in your organization there is a dashboard with a green checkmark on it, and somebody is about to read that checkmark as the software works. That reading is not slightly optimistic; it is a category error. The checkmark is the conjunction of several thousand small, specific, human-authored claims, each of which held on one run under one arranged set of conditions. Everything it says is true. Almost nothing it implies is.
This module builds the precision that the rest of the guide stands on: what a single test asserts, what a suite of them adds up to, and what a test worth having looks like structurally — three phases, one reason to fail, and behavior rather than machinery on the assertion side. If you get this module right, every later question in testing — how many, at what level, with how much mocking, chasing what coverage number — becomes the same question about where to spend a budget you can already price.
A test is a claim about behavior, made executable and repeatable. That definition does real work, so take it word by word. Claim about behavior: it says something falsifiable about what the system does when observed from outside — not about how the system is built. Executable: a machine can settle the claim without a human's judgment in the loop. Repeatable: the same code and the same inputs produce the same verdict, today and on the ninth run in parallel on a CI box in another timezone. Drop repeatability and you no longer have a test; you have an observation.
Now read what a pass actually asserts, in its full and narrow form: under exactly these arranged conditions, this one observed behavior matched this one expectation, on this run. Four qualifiers, all load-bearing. It does not say the behavior is correct for conditions you did not arrange. It does not say the expectation is the right expectation — a test can faithfully pin a bug that the author believed was a feature. It does not say anything at all about the code paths it never entered. And on a nondeterministic test it does not even reliably say what it said last time.
Here is the running example this guide uses throughout. You maintain a wine-cellar application: a user's cellar holds lots (a wine, a vintage, a quantity in bottles), a price feed publishes market quotes, and the system values the cellar. The first test:
def test_value_cellar_prices_per_bottle():
# arrange
cellar = Cellar(lots=[Lot(wine_id="chx-2016", vintage=2016, bottles=6)])
feed = StubPriceFeed({("chx-2016", 2016): Quote(per_case_cents=480000)})
# act
result = value_cellar(cellar, feed)
# assert
assert result.total_cents == 240000The claim is exactly this: given one lot of six bottles and a feed quoting 4,800.00 per twelve-bottle case — 400.00 a bottle — the valuation is 2,400.00. Nothing about empty cellars, nothing about multi-vintage wines, nothing about what happens when the feed times out. One claim, filed.
A passing test is not evidence that code is correct. It is evidence that one specific, human-chosen claim about behavior was not contradicted on this run. Every inference you draw beyond that sentence is something you added, not something the test gave you.
People describe a test suite as a safety net, and the metaphor is doing quiet damage. A net is characterized by its coverage of the space beneath you: it either extends under a region or it doesn't, and you can see the difference. A test suite is characterized by nothing so visible. It is a set of claims somebody chose to file, on the questions somebody thought to ask, in the time somebody had. The regions no one thought about produce no signal at all — not a warning, not a gap in the weave. They produce green, exactly like the regions that were tested exhaustively.
So: a suite is a portfolio of bets about what might break. Green means the bets you placed came up good. It is silent — perfectly, indistinguishably silent — about the risks you never bet on. This reframing is the whole guide in one move, because it converts every testing question into a portfolio question: which risks deserve a bet, how much does each bet cost, and what does a winning bet actually pay out in confidence?
You already know this shape from litigation. A clean discovery record is not innocence; it is the absence of contrary evidence on the questions actually propounded. Nobody in a deposition confuses "we found nothing" with "there is nothing" — they ask what was requested, how it was scoped, and what a competent adversary would have asked instead. Read a test suite with the same reflex: not is it green, but what was propounded.
A readable test has three phases with distinct jobs, and mixing them is the first thing that makes a suite unmaintainable. Arrange builds the world: the inputs, the doubled dependencies, the fixture state. Act pokes that world exactly once — one call to the system under test. Assert checks one outcome. Reading a test should therefore be reading a sentence: given this, when that, then this.
The structure is not aesthetic. It buys three concrete things. First, the arrange block is a written specification of everything the behavior depends on — if you cannot enumerate the dependencies, you cannot reason about the claim's scope. Second, a single act localizes causation: when the assert fails, exactly one operation produced the failure. Third, the shape is diagnosable by strangers, including the future you at 23:40 during an incident.
The arrange block is also a design instrument. A test whose arrange runs to forty lines — five doubles, a database, an environment variable, a frozen clock — is not a badly written test. It is an accurate report that the system under test has too many dependencies. Test pain is design feedback, and it is the cheapest design feedback you will ever get, because it arrives before the code has users. The usual mistake is to treat that pain as a testing problem and reach for more tooling, when the code is telling you to split the unit.
Some tests genuinely need a second act — the "do it twice" shape that checks idempotency, for example. That's fine when the repetition is the behavior under test. The rule is one act per claim, not one call per test.
A test should have exactly one reason to fail. Not one assertion statement necessarily — one reason: one coherent claim whose falsity is the only thing the red can mean. The moment a test asserts three unrelated things, its failure message stops being an answer and becomes a question, because the first assertion to fail aborts the rest and hides them. You learn that something in a bundle is wrong, then pay diagnosis time to find out what.
Diagnosis time is where suites actually get expensive, and it is systematically underestimated because it doesn't appear on any dashboard. Writing a test is a one-time cost measured in minutes. Running it is a recurring cost measured in seconds. Diagnosing its failures is a recurring cost measured in engineer-hours under time pressure, and it is the cost that determines whether people trust the suite or route around it. Failure localization is the economic point of small tests — not purity.
Compare the two versions of the same cellar test below. The bad one bundles the valuation math, the ordering of a returned list, and a logging side effect. When it goes red, it has told you almost nothing: three plausible culprits, one message, and a stack trace pointing at whichever assertion happened to be first.
Tests are often praised as the documentation that can't go stale, and that is true under exactly one condition: that they describe observable behavior. A test asserting what the code does stays true across every rewrite that preserves the behavior, which is precisely what makes it documentation. A test asserting how — call order, private state, the shape of an intermediate data structure, which collaborator was invoked how many times — documents the current implementation, and its truth expires the moment somebody improves the code.
This is implementation coupling, and its signature in the wild is unmistakable: a refactor that changes no behavior turns a wall of tests red. Teams read that as evidence the refactor was risky. It is nearly always evidence that the tests were coupled. The tell is diagnostic: if you can rewrite the internals, keep every externally observable behavior identical, and still break the tests, the tests were never testing behavior. Module 3 shows the mechanism by which this happens most often — mocking a boundary you own — and how to undo it.
The worst payoff is asymmetric. Implementation-coupled tests fail loudly on safe changes and stay silent on genuine bugs, because a bug that leaves the call graph intact leaves the assertions intact. You get maximal noise and minimal signal, which is a bad portfolio at any price.
Symptom: a test asserts that your ORM saved a row, that your web framework routed /valuation to the handler you registered, or that a serialization library round-trips a dataclass. Why it happens: those tests are easy to write and they make the count go up. What's wrong: you are testing someone else's code, which has its own suite, and pinning your own code to that library's current API in the process. Corrective: assert on your logic at the seam — that the handler returns the documented valuation for a known cellar — and let the framework's maintainers test the framework.
Behavior: assert response.json()["valuation_cents"] == 240000 — survives switching ORMs, renaming the repository, extracting a helper. Implementation: assert repo.fetch_prices.call_args[0][0] == ["chx-2016"] — breaks when you batch the fetch, change the argument to a set, or pass the lot objects instead of ids, none of which the user can observe. The second one also passes happily while the SQL underneath it returns duplicated rows, which is the actual bug waiting in module 4.
The test pyramid is the most cited and least understood picture in software testing. Cited, because it is easy to draw and easy to turn into a ratio. Misunderstood, because it is usually taught as a rule about proportions when it is actually the conclusion of an argument about prices — and a conclusion whose premises change from codebase to codebase and from year to year.
This module rebuilds the argument from underneath. Three levels defined by what real machinery each wires in; three cost axes that make the levels genuinely different purchases; and a shape that falls out of the pricing rather than preceding it. By the end you should be able to do something more useful than recite a ratio: you should be able to look at a suite, price what it spends, and say which expenditure is mispriced — including in the direction the pyramid does not predict.
The levels are defined by one question: how much real machinery does the test wire in? A unit test wires in none — the code under test and in-memory values, no database, no network, no filesystem, no clock it doesn't control. An integration test wires in some real machinery, deliberately chosen: a real Postgres, real SQL, real serialization, real config loading. An end-to-end test wires in everything and drives the system the way a client does — through the deployed HTTP surface, holding no privileged handle on the internals.
Note what this definition does not depend on: the directory the file lives in, the framework that runs it, or how many classes it touches. A test in tests/unit/ that opens a socket is an integration test that has been misfiled, and it will behave like one — slow, order-sensitive, occasionally red for reasons having nothing to do with your code. Most arguments about "is this a unit test?" dissolve once the participants agree to answer by machinery rather than by folder.
Teams waste real time arguing whether a unit is a function, a class, or a module. The productive definition is a unit of behavior, not a unit of code: the smallest thing you can describe as a claim a caller would care about. Valuing a cellar is a unit of behavior even if it touches four classes; a private helper's return type is a unit of code and probably deserves no test of its own.
Each level is a purchase, and every purchase has a price and a payout. The price has three components, and teams routinely notice only the second.
Cost to write. A unit test for the valuation math is a fixture and a call — minutes. An integration test needs a real database, schema migrations, seeded data, and cleanup — an afternoon the first time, then minutes once the harness exists. An end-to-end test needs the whole system deployable, an authenticated client, and test data that survives a full request path — days of harness work, then hours per test.
Time to run. Roughly: a unit test is a millisecond, an integration test is tens to hundreds of milliseconds, an end-to-end test is seconds to tens of seconds. Multiply by the count and you get the number that actually governs behavior — the feedback latency that decides whether engineers run the suite before pushing or after being paged.
Cost to diagnose. This is the one that gets forgotten and it dominates. A red unit test names the function. A red integration test names a subsystem: it could be your SQL, the schema, the seed data, or the config. A red end-to-end test names the system — the failure could be anywhere along a path crossing every layer, plus the network, plus the environment. Diagnosis cost scales with the surface the test spans, and it is paid in engineer-hours under time pressure.
Against those prices, the payout: breadth of confidence per pass. A passing unit test tells you one function's logic is right and nothing about whether it is wired to anything. A passing end-to-end test tells you an entire user-visible path worked, wiring included — a genuinely broader claim. That is the trade the whole pyramid encodes: precision and cheapness on one end, breadth and expense on the other.
Now the pyramid falls out with no appeal to virtue at all. You have a finite confidence budget. At the cheap end, claims are precise, fast, and diagnose themselves, so you can afford thousands and each red one answers its own question. At the expensive end, claims are broad and only broad claims can catch whole classes of bug — but each costs enough that a hundred of them reshapes your engineering day. Rational allocation buys many cheap precise bets and a few expensive broad ones. That is the pyramid: not a proportion handed down, but the answer to an optimization whose inputs are your prices.
Which means the shape moves when the prices move — and this is the part the ratio-reciters cannot handle. Containerized databases and disposable schemas dropped the cost of an integration test by roughly an order of magnitude over the last decade: what used to require a shared staging database and a cleanup crew is now a fixture that spins a Postgres in two seconds. When a price falls that far, the optimum shifts toward that level, and a suite with a fat integration middle is not a violation of the pyramid — it is the pyramid's argument, run on current prices.
This is also all that the "testing trophy" and its cousins are saying. They are re-pricings of the same cost/confidence argument for stacks where integration got cheap and unit tests of thin glue code buy little. Treat them as inputs to the same calculation, not as rival doctrines to enlist under. If a shape is defended by naming its author rather than by pricing its levels, the argument has already gone wrong.
The pyramid is a conclusion, not a premise. Anyone who states a ratio without stating the prices it was derived from has skipped the argument — and will keep the ratio after the prices that justified it have changed.
The inverted suite — a broad top of slow end-to-end tests, a thin middle, almost nothing at the base — is the pyramid's argument run backwards, and it almost never arrives through ideology. It arrives honestly. A team inherits code with no seams: business logic lives inside request handlers, database calls are interleaved with rules, and nothing can be called without standing up the world. The only place to attach a test is the outside, so tests get written at the outside. Every individual decision was locally correct.
The bill comes due exactly where the pricing predicts. Feedback latency: the suite takes forty minutes, so nobody runs it locally and defects are found by CI after the context switch — the same bug that cost fifteen seconds to fix at your desk now costs an hour of re-immersion. Diagnosis by archaeology: a red end-to-end test names the system, so triage begins with reading logs across four services to find which layer moved. Maximal flake exposure: every end-to-end test crosses the network, real time, and shared state — the three richest sources of nondeterminism (module 5) — so the level with the highest diagnosis cost also fails most often for reasons unrelated to your code. That combination is what teaches teams to add retries, and retries are how a suite's signal dies.
Symptom: a suite that takes tens of minutes, a "flaky" label applied to a rotating cast of end-to-end tests, and nobody running tests locally before pushing. Corrective: don't start by deleting end-to-end tests — you'll lose real coverage of the wiring. Start by creating seams: extract the business logic out of the handlers so it can be called directly, add unit tests for the logic you just freed, then retire the end-to-end tests whose only remaining claim is now covered more cheaply below. The count goes down and the confidence goes up, which is the pricing argument paying out.
Module 2 priced the levels. This module is about the mechanism that makes the cheap level cheap: replacing real dependencies with stand-ins. Every fast unit test in your suite exists because somebody swapped a database, a clock, or an HTTP client for something that answers instantly and identically every time.
That swap is not free, and the invoice arrives later. A test double buys speed and determinism by encoding an assumption about how the real dependency behaves — and it charges you coupling: to the dependency's contract, and, if you place it badly, to your own code's current internal structure. This module fixes the vocabulary precisely (four species, used with that precision for the rest of the guide), doubles the same dependency three ways to show what each purchase buys, and then dissects the single most common self-inflicted wound in testing: mocking a boundary you own.
You replace a dependency for two reasons, and only two are legitimate. Speed: a real HTTP call to the price feed takes 200 ms, so a thousand valuation tests would take three and a half minutes instead of one second. Determinism: the real feed returns today's market price, which changes, so a test asserting an exact valuation against it fails for reasons that have nothing to do with your code — the repeatability failure from module 1.
What you pay is fidelity. Every double is a small model of the real thing, and the model encodes a bet: I understand this dependency's contract well enough that a test passing against my model would also pass against reality. Sometimes that bet is trivially safe. Often it isn't: the real feed returns null for delisted wines and your stub never does; the real Postgres enforces a unique constraint your in-memory fake doesn't; the real API returns 429 with a Retry-After header in seconds, and your double returns it in milliseconds.
Write it down as an inequality worth remembering: a green unit test proves your code is correct given your model of the world. When the model is wrong, the test is green and the system is broken, and no amount of adding more unit tests will find it — they all share the same model. This is precisely the gap integration tests exist to close (module 4), and it is why a double is never a substitute for a real-dependency test; it is a trade that creates a debt which a real-dependency test is the only instrument that can pay.
Every double encodes a falsifiable assumption about the real dependency's contract. Name that assumption out loud when you write the double, then name the test that checks it against the real thing. A double whose assumption nothing ever verifies is an unaudited claim sitting under every test that uses it.
Teams call every double "a mock," and the imprecision is not pedantic — it hides what a test verifies. Four species, defined here and used with this precision for the remainder of the guide.
A stub returns canned answers and verifies nothing. It exists to let the act happen: StubPriceFeed({("chx-2016", 2016): Quote(480000)}) answers one question one way, and the test's assertion is about the return value of your code, never about the stub.
A fake is a working lightweight implementation that honors the real contract — an in-memory repository with real add, query, and dedupe semantics. It has behavior, so it supports multi-step tests (write, then read back) that a stub cannot, and its fidelity risk is that its semantics drift from the real component's.
A mock is a double with expectations: it asserts the interactions it receives. The test's claim moves from "what did the code return" to "how did the code talk to its collaborator." That is a legitimate claim exactly when the interaction is the behavior — a retry, a payment call, an audit emission — and a trap otherwise.
A spy records calls for you to assert on after the act, rather than declaring expectations before it. Mechanically similar to a mock, temperamentally different: assertions live in the assert block where you can see them, which usually reads better.
| Species | What it verifies | Coupling it incurs | Reach for it when |
|---|---|---|---|
| Stub | Nothing — it only supplies inputs | To the dependency's return shape only | The claim is about your code's output |
| Fake | Nothing directly; enables multi-step behavior | To the dependency's semantics (drift risk) | The test needs the dependency to behave, not just answer |
| Mock | The interactions your code performed | To the current call graph — the strongest coupling of the four | The interaction is the behavior under test |
| Spy | Interactions, asserted after the fact | Same as mock, but assertions are visible in the assert block | You need interaction evidence with readable failures |
"We mocked the repository" and "we faked the repository" describe different tests with different failure modes. The first probably asserts call arguments and will break on your next refactor; the second probably asserts a returned value and will survive it. If the team uses one word for both, nobody can tell which test they have without reading it.
Take one dependency — the external price feed — and three genuinely different test goals. The species follows from the goal, not from habit.
Goal: the valuation math is right. → Stub. The test needs the feed to answer, and asserts on result.total_cents. It should be indifferent to how many times the feed was consulted, in what order, and with what batching, because none of that is the behavior under test. Encoded assumption: the feed returns per-case cents as an integer for a known (wine, vintage) pair. Checked by: a contract test against the sandbox feed.
Goal: the repository dedupes lots by (wine_id, vintage). → Fake. The test writes two lots and reads back, so it needs the double to behave, not just answer. A stub cannot express "what you wrote is what you get back." Encoded assumption: the store's uniqueness semantics are (wine_id, vintage). Checked by: the integration test against real Postgres in module 4 — and this is exactly the assumption that turns out to be wrong there.
Goal: the client retries once on HTTP 429. → Mock. Here the interaction is the behavior: there is no return value that distinguishes "retried and succeeded" from "succeeded first time." So you assert the call sequence — first call returns 429, second returns 200, exactly two calls — and that assertion is legitimate because retrying is the specified behavior, not an implementation detail. Encoded assumption: the feed signals rate limiting with 429 and honors a retry.
Here is the rule, and it does more work than any other rule in this guide: double at boundaries you don't own; use the real thing (or a fake with a contract test) at boundaries you do. A boundary you don't own is code whose behavior you cannot change and whose failure modes you must model anyway — the external price feed, the payment processor, the identity provider. A boundary you own is your repository, your service layer, your domain objects. Doubling the first is prudence. Doubling the second welds the test to your current call graph.
Watch the mechanism on the running example. Somebody writes this, in good faith, to keep the valuation test fast:
# the over-mocked test
def test_value_cellar_fetches_prices():
repo = Mock(spec=PriceRepository)
repo.fetch_prices.return_value = {("chx-2016", 2016): 480000}
cellar = Cellar(lots=[Lot("chx-2016", 2016, bottles=6)])
value_cellar(cellar, repo)
repo.fetch_prices.assert_called_once_with(["chx-2016"])Read what this test claims: value_cellar called fetch_prices once with a list containing one wine id. That is a claim about your code's internal call graph and about nothing else. It does not assert the valuation. It does not assert the returned rows are right. And the actual bug — the repository's SQL joins prices on wine_id alone while the prices table keys on (wine_id, vintage), so a wine with three vintages in the catalog returns three rows per lot and triples the cellar's value — sails through untouched, because the mock returned exactly the dictionary the author imagined the real query would return. The mock verified the author's assumption, not the world.
Then the second half of the wound. A teammate extracts a helper so prices are fetched per lot batch, passing lot objects instead of ids. Behavior is identical; the number the user sees is unchanged. Nine tests go red on assert_called_once_with. Maximum noise on a safe change, zero signal on the live bug — the asymmetry from module 1, now with a named mechanism.
The refactor is unglamorous: delete the interaction assertion, use a fake repository with real dedupe semantics, and assert the valuation.
# the refactored test
def test_value_cellar_prices_each_bottle_in_the_lot():
repo = InMemoryPriceRepository()
repo.add(wine_id="chx-2016", vintage=2016, per_case_cents=480000)
cellar = Cellar(lots=[Lot("chx-2016", 2016, bottles=6)])
result = value_cellar(cellar, repo)
assert result.total_cents == 240000This survives the extract-helper refactor, because it never knew how prices were fetched. It still cannot catch the SQL bug — no test with a doubled repository can, by construction — but it has stopped pretending to. The claim that the real query returns the right rows is now visibly unfiled, which is exactly the state module 4 goes on to fix with a real database.
Symptom: tests full of assert_called_with, a mock for every collaborator, and a suite that reddens on every refactor while production incidents keep arriving from code that is "fully tested." Corrective: for each mock, ask what behavior would be wrong if the assertion failed. If you can only answer "the code would be structured differently," the assertion is pinning structure — replace the mock with a fake or the real component and assert the outcome instead. Keep interaction assertions only where the interaction is the specified behavior, as with the 429 retry.
Module 3 ended with a debt: every double encodes an assumption, and nothing so far has checked any of those assumptions against reality. This module collects that debt. Integration tests wire in real components to test the assumptions your doubles encode; end-to-end tests drive the system as a client to test the one thing no internal test can see — whether what you actually serve matches what you promised.
The demonstration is the running example, completed. One capability, three bugs, three levels — and the discipline of this module is that each bug must be mechanically invisible to the level below it. Not merely unlikely to be caught. Invisible by construction, for a reason you can state in a sentence. That is the standard for justifying an expensive test: name the claim only this level can make.
Units prove the parts. Integration tests prove the wiring — and "wiring" is a bigger category than it sounds: real SQL against a real schema, real serialization and deserialization, real config loading, real transaction boundaries, real constraint enforcement. Every one of those is code you did not write, behaving according to rules you assumed rather than verified.
Take the bug that lives here. The cellar's price repository fetches quotes with a query that joins on wine_id:
SELECT l.wine_id, l.vintage, l.bottles, p.per_case_cents
FROM lots l
JOIN prices p ON p.wine_id = l.wine_id
WHERE l.cellar_id = %sThe prices table, however, keys on (wine_id, vintage) — a wine has a different market price in each vintage year, which is the entire point of the domain. Chateau X 2016 and Chateau X 2019 are two rows in prices. So for any wine with three vintages priced, this join returns three rows per lot, and the valuation sums all of them: a cellar heavy in multi-vintage producers reports roughly triple its real value. It is a one-word bug — the missing AND p.vintage = l.vintage — and it is worth real money to the user.
Now the mechanical invisibility. No unit test can catch this, and not because unit tests are careless. The bug lives in the SQL string, and a unit test by definition wires in no database — so it either mocks the repository (in which case, per module 3, it asserts your assumption about the returned rows) or fakes it (in which case the fake implements the semantics you intended, dedup and all). Both doubles are built from the same mental model that produced the wrong join. You cannot catch a modeling error with a model you built. The only instrument that helps is the real component.
Spin a disposable Postgres, run the real migrations, insert one cellar with one lot of Chateau X 2016 (six bottles) and three price rows for that wine across vintages 2016, 2018, and 2019 (480000, 540000, and 600000 cents per case), then call the real repository. Assert the returned valuation is 240000; against the buggy join it comes back 810000. The test is roughly 90 ms, and it is the only cheap thing in your suite that could ever have found this.
An end-to-end test drives the system the way a client does: over the real transport, through the real serialization, holding no privileged handle on anything inside. That restriction is the entire source of its unique value. It is the only level that reads what you actually serve.
The third bug. Your published API contract documents the valuation endpoint as returning an integer field valuation_cents. The handler, after a refactor, returns {"valuation": 12408.5} — float, dollars, different field name. Every internal test still passes: the valuation logic is right, the repository is right, the integration test asserts on the repository's return value and never touches the HTTP layer. The number is correct. The contract is broken, and mobile clients parsing valuation_cents as an integer will either crash or silently read a missing field as zero.
This is contract drift, and the invisibility is again mechanical: no test below the client level reads the serialized response body. The integration test holds a Python object; the serializer runs after it. Only a test that makes an actual request and parses an actual response occupies the position from which the drift is observable.
Nº 07 argues that the published contract is the product: what you documented is what integrators built against, and unilaterally changing it is a breaking change no matter how correct the new shape is. This module supplies the enforcement mechanism. An end-to-end test written against the documented schema — not against the current handler's output — is how that contract stops being a document and becomes an executable claim. If your e2e assertions are generated from the same code that produces the response, you have re-created the verification gap module 8 is about.
These levels are slower and less deterministic than unit tests, and it is important to say why precisely, because the reason determines what you should and should not accept.
They are slow for structural reasons: real I/O has latency you cannot optimize away, a database round trip is orders of magnitude slower than a function call, and an end-to-end test additionally pays for process startup, network hops, and often authentication. No amount of engineering discipline turns a 12-second end-to-end test into a 1-millisecond one. That is physics, and module 2 already priced it.
They are less deterministic for structural reasons too: they touch real time, real ordering, real shared state, and the network — the four flake sources module 5 anatomizes. The exposure is inherent to the level.
Now the crucial distinction, because teams routinely collapse it. Inherent exposure to nondeterminism is not the same as tolerable flakiness. "End-to-end tests are flaky by nature" is used to justify retries and shrugs, and that inference is invalid. The correct reading is the opposite: because this level is maximally exposed, it is the level where determinism discipline has to be strongest — the injected clock, the per-test data, the explicit waits on conditions rather than sleeps. A flaky end-to-end test is not the level expressing its nature. It is a bug at the level where bugs are most expensive to diagnose.
When someone says "it's an e2e test, they're just flaky," ask which of the four sources — clock, ordering, shared state, or concurrency — is producing it. There is always an answer, and the question converts a shrug into a bug report. If nobody can name the source, that is itself the finding: nobody has looked.
Assemble the running example. One capability — value a cellar from live prices — three bugs, each caught at exactly one level and invisible below it.
The unit-level bug: per-case quotes against per-bottle quantities. The feed quotes 480000 cents per twelve-bottle case; the valuation multiplies the quote by the number of bottles. A six-bottle lot valued at 2880000 instead of 240000 — a twelvefold error on the per-bottle rate. Caught by a unit test with a stubbed feed in about a millisecond. Nothing above needs to be involved; an end-to-end test would also catch it, at ten thousand times the cost and with a failure message implicating the whole system.
The integration-level bug: the vintage join. Prices joined on wine_id alone, duplicating rows for multi-vintage wines. Invisible to units because every unit doubles the repository with the semantics the author intended. Caught against a real disposable Postgres with real migrations.
The end-to-end bug: the cents/dollars contract drift. Documented valuation_cents as an integer; served valuation as float dollars. Invisible to the integration test, which asserts on an object before serialization. Caught by a client-driving test asserting against the documented schema.
Read the pattern rather than the list. Each level's unique value comes from the machinery it wires in that the level below excludes by definition. That is why the justification for an expensive test is never "it's more realistic" — it is "here is the claim, and here is why nothing cheaper is positioned to make it." A suite built that way is expensive only where it must be.
A flaky test passes and fails on identical code. Teams describe these as weather — unlucky, atmospheric, something to wait out — and that vocabulary is the whole problem, because weather is not anybody's fault and has no root cause. A flake is a bug. Specifically it is a bug in the test's determinism, which is one of the two properties that made it a test at all (module 1), and like any bug it has a cause you can find.
This module does two jobs that turn out to be one job. It teaches the autopsy — how to take an intermittent failure down to a named root cause — and it teaches how to test the genuinely hard things: time, data, randomness, and services you don't control. Those are the same subject, because the things that are hard to test are exactly the things that produce flakes when tested carelessly. And running underneath both is the argument that matters most to anyone who has to defend an engineering budget: one tolerated flake does not cost you one test. It costs you the signal of the entire suite.
Here is the case. test_cellar_valuation_uses_current_window_prices passes on every developer machine and fails roughly one CI run in ten. The failure message varies: sometimes the valuation is 810000 instead of 240000, sometimes it is 0. It has been red four times this month and green on rerun each time. Somebody has added it to a document called "known flaky tests."
The discipline is the same one you would apply to any intermittent production bug, and it has three moves. Reproduce: make the failure happen on demand, which for flakes means finding the conditions rather than the input — run the test 200 times, run it in parallel, run it in the CI container's timezone, run it in a shuffled order. Isolate: change one condition at a time until the failure rate moves. Bisect: not over commits (the code hasn't changed) but over the environment — which variable, when flipped, flips the outcome?
Applied here, two things fall out quickly. Running the test alone in a loop 500 times: always green. Running the full class in parallel: red about 12% of the time, and always with 810000. Running the test alone under TZ=UTC after 16:00 local: red, deterministically, with 0. Two independent root causes wearing one failure name — which is common, and which is why "retry until green" is so seductive: a single retry usually clears both, and neither ever gets diagnosed.
A flaky test is not a test that sometimes fails. It is a test whose outcome depends on a variable it does not control and did not declare. Find the undeclared variable and you have found the bug — and the bug is usually in the test, but not always, because production code that depends on undeclared variables is exactly how race conditions reach customers.
Every flake traces to one of four families. The taxonomy is worth memorizing because it converts "it's flaky" into a search with four branches.
Time. Wall-clock reads, timezone assumptions, date boundaries, expiry windows, and anything computed relative to now(). These fail at midnight, at month ends, on leap days, during daylight-saving transitions, and whenever CI runs in UTC while developers run in local time.
Ordering. Tests that pass only in the order the author happened to run them — because an earlier test left state behind, or created a record a later test assumes exists. Shuffling the suite reveals these instantly, which is why running shuffled is worth doing deliberately.
Shared state. Class-scoped or session-scoped fixtures, module-level caches, singletons, the same database row, the same temp file, the same port. The efficiency argument for sharing is real and the coupling it creates is what turns two innocent tests into an intermittent failure.
Concurrency and the network. Races between threads or processes, tests that wait a fixed duration for an async result, timeouts against real services, and port collisions. The signature is failure rate that changes with machine load — which is why these appear in CI and not on a fast laptop.
Now the autopsy's two findings, named. Root cause one — shared state under parallel ordering: the test class uses a class-scoped fixture that seeds one cellar row, and a sibling test, test_add_lot_to_cellar, adds a second lot to that same cellar. Run sequentially in file order, the valuation test runs first and sees one lot. Run in parallel, the sibling sometimes commits first, and the valuation test sees two lots — hence 810000. The fixture was shared "for speed," saving about 40 ms per class. Root cause two — the clock: the valuation selects prices in the current pricing window, computed as date.today() in local time. The CI container runs UTC. After 16:00 Pacific, UTC's date has already rolled over, the window query selects a day with no seeded prices, and the valuation comes back 0. The test was correct in one timezone and wrong in every other — which means it was always wrong, and only ever ran somewhere forgiving.
Now the argument that has to be made in expected-value terms, because "flakes are bad" loses every prioritization meeting to a feature.
A test suite produces exactly one thing: information. Specifically, it produces the inference red implies broken, and every decision built on the suite — merge, deploy, roll back — is a purchase of that inference. So price it. Suppose a red build is worth investigating if red means broken most of the time. With zero flakes, red means broken essentially always, and the rational response to red is to stop and read. Add five tolerated flakes each failing one run in ten, and roughly 40% of your runs contain at least one spurious red. The engineer facing a red build now faces a coin flip, and the cheapest correct-on-average action is to hit rerun. That action is individually rational and collectively fatal, because the day a rerun clears a real intermittent bug — a genuine race in production code — nobody notices, and the suite has now actively concealed a defect it detected.
Note the structure of the loss. You did not lose the flaky tests' signal. You lost the signal of every test in the suite, including the thousand that are perfectly deterministic, because the inference you were buying was about the build's color, not about any individual test. That is why the correct policy target is a flake rate near zero rather than "an acceptable number of flakes," and why quarantining a flake — moving it out of the merge-blocking suite the same day, with a ticket and an owner — is a legitimate move while retrying it in place is not. Quarantine removes the noise and preserves the knowledge that a claim is currently unfiled. Retry hides both.
Symptom: retries: 3 in the CI config, a "known flaky" document, and a team norm of rerunning red builds before reading them. What it actually buys: it converts a loud intermittent bug into a silent one and spends suite-wide trust to do it. It also destroys the evidence — an auto-retried failure usually discards the failing run's logs, which is precisely the artifact an autopsy needs. Corrective: treat every flake as a bug with an owner and a root cause from the four families; quarantine out of the blocking suite within a day if it can't be fixed immediately, and track quarantine size as a number somebody is accountable for.
You have seen this exact failure in security operations and GRC. An alert that fires falsely gets tuned out, then routed to a folder, then disabled — and a control whose alerts nobody acts on is not a control, whatever the policy document says. Auditors ask for evidence that alerts were investigated for precisely this reason: the operating effectiveness of a detective control lives entirely in the response, not in the configuration. A flaky test is a detective control with a false-positive rate high enough to have trained its operators to ignore it, and "we rerun the build" is the engineering equivalent of an alert queue nobody reads.
The cures are all the same move: find the undeclared variable and turn it into a declared input. That injection point is a seam, and building seams is most of what makes code testable.
Time. Stop reading the clock inside business logic. Pass it in — a clock dependency, a now parameter, an injected provider — and in tests freeze it at a fixed instant. Two rules make this actually work: freeze at an interesting instant, not a convenient one (23:59:30 on the last day of a month in a non-UTC zone finds more bugs than noon on a Tuesday), and never assert against a value computed from the current time, which just moves the nondeterminism into your assertion. Fixing the autopsy's clock bug: value_cellar(cellar, repo, clock=FrozenClock("2026-03-14T09:00:00-07:00")), and the pricing window is derived from the injected instant in an explicit timezone.
Data. Every test owns its data. The reliable pattern is a per-test transaction rolled back at teardown — fast, complete, and impossible to leak — with factory helpers building exactly the rows a test needs rather than a shared seeded world. Fixing the autopsy's shared-fixture bug: the class-scoped cellar becomes a per-test factory call inside a transaction. The measured cost of that change is about 40 ms per class. The measured benefit is that the test's outcome no longer depends on which sibling committed first.
Randomness. Seed it explicitly and log the seed, so a failure is reproducible from its own output. An unseeded generator in a test is an undeclared input by definition.
Concurrency. Never sleep() to wait for an async result. A sleep is a bet about the scheduler: too short and it flakes, too long and it taxes every run forever, and it changes the odds of the race without removing it. Wait on the condition instead — poll for the state you need with a timeout, or better, expose a completion signal the test can await deterministically.
Then prove it. Determinism is a claim, so give it a criterion: 500 consecutive passes under randomized order with parallelism at 8, in the CI container's timezone. Both autopsy fixes clear that bar; before the fixes the same protocol reproduced each failure within 40 runs. A fix without a proof criterion is a hope.
The hardest determinism problem is a dependency somebody else operates: the price feed, a payment processor, an identity provider. It has downtime you don't schedule, rate limits you share with other consumers, and data that changes without notice. There are exactly three honest treatments, and the choice is economics again — module 2's argument, applied to a single dependency.
A fake honoring the contract. Fastest and fully deterministic; costs you fidelity, and the fidelity debt must be paid by a contract test against the real thing (module 3). Correct default for the merge-blocking suite.
A recorded interaction, replayed. Capture real request/response pairs once, replay them thereafter. Higher fidelity than a hand-built fake because the recording came from reality — and it silently ages, because the vendor's behavior drifts while your cassette does not. Re-record on a schedule or it becomes a fake with false authority.
A real call, quarantined. Highest fidelity, genuinely nondeterministic; belongs outside the merge-blocking suite — a scheduled job whose failures page an owner rather than block a developer's merge. This is where your contract tests live. The key discipline: its failures must mean something to somebody, or you have built a cron job that produces ignored red.
Running tests against real services makes you an API consumer with a retry problem, which is Nº 07's territory. Setup and teardown must be idempotent — the same test run twice cannot leave two orders, two accounts, or two subscriptions behind — and every test needs a unique key namespace so concurrent runs don't collide on the vendor's side. Nº 07's idempotency-key mechanism is what makes a re-run test suite safe against a real system; without it, your third rerun of the afternoon is creating production data.
Vendor sandboxes are shared, mutable, and periodically reset — three properties that make them a shared-state flake source (horseman three) wearing a vendor's logo. They are fine for scheduled contract tests and poor for anything that blocks a merge. If a sandbox failure can stop your team from shipping, you have outsourced your release gate to someone with no obligation to you.
Coverage is the only number most organizations have about their test suites, which is why it ends up carrying weight it cannot bear. It appears in board decks, in engineering OKRs, and in merge gates, and in every one of those places it is being read as a measure of verification. It is not one. It is a measure of execution.
This module states exactly what line and branch coverage measure, builds a fully covered function with a live production bug in it, and then works through what happens organizationally when the number becomes a target — a process reliable enough to predict quarter by quarter. It closes with the honest use, which is real: coverage is an excellent detector of what you have not tested, and a worthless certificate of what you have.
Line coverage is the percentage of executable lines that ran at least once while the test suite ran. Branch coverage is the percentage of decision outcomes — both sides of each if, each loop's zero-iteration and at-least-one-iteration cases — that were exercised. Branch is strictly stronger, because a line containing a condition can execute with only one outcome taken.
Read those definitions for what they do not contain. Neither mentions assertions. Neither mentions inputs. A coverage tool watches the interpreter and records which lines the program counter visited; it has no opinion about whether your test checked anything afterward, and no concept of input classes — an empty list, a null, a negative number, a leap day — because it counts lines, not the space those lines were exercised over.
That gives coverage a precise and genuinely useful asymmetry. Zero coverage is proof. A line that never executed under test is definitively untested, and no argument can rescue it — this is a real fact worth having, and it is why coverage tooling belongs in your workflow. Full coverage is nothing. A line that executed may have been asserted on exhaustively, or may have run under a test that checked nothing at all. The metric cannot distinguish those cases, and the distinction is the whole question.
Coverage is a competent detector of untested code and an incompetent certifier of tested code. The asymmetry is not a limitation to work around; it is what the measurement is. Every misuse of coverage comes from reading the second direction as if it were as sound as the first.
Here is the module in question, reported at 95% line coverage:
def value_cellar(cellar, repo, clock):
prices = repo.prices_in_window(clock.now())
total = 0
for lot in cellar.lots:
per_bottle = prices[lot.wine_id, lot.vintage] // 12
total += per_bottle * lot.bottles
bottle_count = sum(lot.bottles for lot in cellar.lots)
return Valuation(
total_cents=total,
average_per_bottle_cents=total // bottle_count,
)And here is the test that covers it:
def test_value_cellar_returns_a_valuation():
result = value_cellar(sample_cellar(), repo, FrozenClock(NOON))
assert result is not NoneEvery line of value_cellar executes. The coverage report is green from top to bottom, including the average_per_bottle_cents line. The assertion is vacuous assertion in its purest form — is not None can only fail if the function raises or returns nothing, which means it tests the absence of a crash and nothing else. The valuation could be negative, zero, twelvefold wrong, or a string, and this test would be green.
Now the live bug. An empty cellar — a real state, since new users have one before adding their first wine — makes bottle_count zero and the final division raises ZeroDivisionError. Every line involved is fully covered. The uncovered thing is not a line; it is an input class, and no line-oriented metric can represent it. In production this surfaces as a 500 on the dashboard endpoint for every new user, from a module whose 95% coverage was cited in the launch review.
is not None, producing a green gutter and a 95% report. Right: the empty-cellar input runs through those same fully covered lines into a division by zero. Coverage counts execution, not input classes and not assertion strength, so both facts are true at once — the module is covered and it is broken.Goodhart's law: when a measure becomes a target, it ceases to be a good measure. Coverage is the canonical software example, and the mechanism is worth spelling out because it is predictable enough to forecast.
Quarter one, the mandate lands: 90% coverage, gated at merge. Engineers are not cynical; they are constrained. The gate blocks work, so the cheapest way to unblock work becomes the norm, and the cheapest ways are all available. Assertion-free tests — call the function, assert is not None, collect the lines. Tests for trivia — getters, constructors, __repr__, generated data classes, all cheap and all high-yield in lines. Exclusion pragmas on the genuinely hard files, since the error-handling paths and the legacy module with no seams are the most expensive lines to cover, so they're the first ones marked no cover.
Quarter two the number reads 91%. Quarter three, 93%. Escaped defects hold flat, then rise — because engineering time went into covering easy lines rather than filing claims about risky behavior, and because the hard code got excluded precisely for being hard. The number is now measuring the team's compliance effort, which is exactly what Goodhart predicts, and the organization has less information about its risk than before the mandate, having replaced "we don't know" with a number that reads like an answer.
This is the design-versus-operating-effectiveness distinction you already enforce in GRC. A SOC 2 auditor testing a control's design asks whether it exists and is capable of working; testing its operating effectiveness means pulling a sample and confirming it actually worked over the period. Coverage-as-gate is a design test dressed as an effectiveness test: it confirms the code was exercised, then gets read as confirming the code was verified. You would not accept "the log pipeline is configured" as evidence that alerts were reviewed — and "the lines executed" is the same claim with the same gap.
Symptom: a coverage gate in CI, a rising percentage, tests whose only assertion is is not None or assert True, test files for data classes, and no cover pragmas concentrated on error handling and legacy modules. Corrective: remove the hard gate and put coverage in the review conversation instead — a diff's newly uncovered lines are a required discussion item, not an automatic block. If a threshold is politically unavoidable, apply it to new code in the diff and pair it with mutation spot-checks (module 8), which cannot be satisfied by assertion-free tests.
None of this makes coverage useless. It makes it a map rather than a certificate, and maps are valuable when read in the direction they're accurate.
Read the uncovered lines of a diff before merging. This is the highest-value use of the tool, and it takes about ninety seconds. For each uncovered line ask one question: what claim is missing here? Usually the answer is "nothing worth filing" — a logging branch, a defensive re-raise. Sometimes the answer is "the entire error path of the payment call," and you have just found an unfiled claim in a risk region that matters. The tool located it; the human priced it.
Treat a drop as a question, not a violation. Coverage falling four points on a diff might mean untested new code, or it might mean the diff deleted a large well-tested module, or added a generated file. Gating on the delta automates a question that needs an answer, which is how you get engineers who write tests to satisfy a number.
Never read a covered line as a verified line. The empty-cellar bug lives at 95%. If you want evidence that covered code is actually verified, the instrument is mutation-shaped — break something deliberately and check the suite notices (module 8) — because that is the only cheap check that responds to assertion strength rather than execution.
The honest summary to give a leader who wants one number: coverage tells you where you certainly have no claims filed, and tells you nothing about the quality of the claims you do have. It is a smoke detector that reports which rooms it isn't installed in.
Every test so far has been example-based: you pick an input, you state the expected output, you file one claim about one point. That is a fine instrument and it has a structural limitation — the claims you file are limited by the inputs you imagined, and the bugs that reach production are disproportionately the ones nobody imagined. The empty cellar in module 6 was not hard to test. It was hard to think of.
This module covers two techniques that widen the portfolio in different directions — properties, which assert a rule across a region of input space and let a machine hunt for counterexamples, and snapshots, which assert that a large output has not changed without asking you to enumerate what it should contain. Then it closes the practice half of the course with test-driven development, weighed honestly: real benefits, real costs, and a choice made per task rather than by allegiance.
An example test pins a point: six bottles at 480000 per case value at 240000. A property-based testing test asserts a rule over a region: for any cellar and any price table, the total valuation equals the sum of the lot valuations. The framework then generates hundreds of inputs from that region, checks the rule on each, and reports any input that breaks it.
The payoff is that generators are indifferent to your imagination. They will produce the empty cellar, the lot with zero bottles, the price of exactly zero, the cellar containing the same wine twice, and the quantity that trips an assumption you never wrote down — the inputs that were never on your list because they were never on anybody's list. Three properties for the valuation, each falsifiable and each catching something real:
Then shrinking, which is what makes the technique usable rather than merely clever. When a generator finds a failure on a randomly built cellar of 47 lots with baroque prices, that input is nearly useless for debugging. The framework automatically reduces it — fewer lots, smaller numbers, simpler strings — while checking the failure persists, and reports the minimal input that still breaks: a single lot with quantity 0. You get a bug report the size of a unit test.
The failure mode of property testing is writing a property that restates the implementation. assert value_cellar(c) == sum(price[l] // 12 * l.bottles for l in c.lots) is not a property; it is the function, re-typed. It passes for every generated input including the ones where the code is wrong, because the code and the "property" make identical mistakes. A real property must be checkable by a route the implementation doesn't take — an algebraic relation, an invariant over the output, a comparison against a slow obviously-correct reference. If you can't state it without transcribing the code, you don't have one yet. This is the same correlated-blind-spot failure module 8 examines when both code and tests come from one author.
Snapshot (approval) testing inverts the usual workflow. Instead of stating what the output should be, you run the code, save its output as a committed artifact, and assert on every future run that the output still matches. The assertion is nothing changed, and the review of what the output should be happens once, in a human's head, when the snapshot is first approved.
Where this earns its keep is large, stable, serialized surfaces where enumerating expectations by hand would be absurd: a valuation report's JSON with sixty fields, a rendered PDF statement, a GraphQL schema, a generated migration. Writing sixty assertions is tedious, unreadable, and no more accurate than a diff. And the failure mode is excellent — when a change alters output, you get a precise diff of exactly what moved, which is often more informative than an assertion message.
Where it rots is the approve-all reflex. A formatting change produces 40 snapshot diffs, an engineer runs the update command, all 40 are re-approved in one keystroke, and the suite has quietly re-defined its expectations as whatever the code currently does. That is the purest form of testing theater in this guide: an assertion that is regenerated from the thing it is supposed to check can never fail meaningfully again. Note that no coverage number moves, no test count drops, and the build is green — the suite looks exactly as healthy as before.
The discipline that keeps it honest is structural, not moral: snapshots are reviewed as code — a diff in a committed snapshot is a diff a reviewer reads, and "update snapshots" alone in a pull request is a review finding. Keep snapshots small enough to read, and never snapshot output containing timestamps, ids, or ordering you don't control, because a snapshot that flakes teaches approve-all faster than anything else.
Test-driven development is the practice of writing a failing test before the code that satisfies it, then writing the minimum code to pass, then refactoring under the test's protection — red-green-refactor. It is argued about with a heat that suggests something other than engineering is at stake, so state the ledger plainly.
What test-first buys. First and most underrated: a test you have watched fail. A test that has never been red is a test whose ability to fail is unverified — you have no evidence it is wired to anything, and module 8's sabotage check exists because of this exact gap. Test-first gives it to you for free, on every test. Second, it applies continuous design pressure toward seams: code written to be called by a test tends to have its dependencies injectable, which is module 5's testability property arriving by default rather than by refactor. Third, it forces you to state the behavior before you build it, which converts vague intentions into a specification early, when changing your mind is cheap.
What test-first costs. It is genuinely awkward when you don't yet know what the behavior should be — exploratory work, spikes, learning an unfamiliar API — where the honest first step is to build something and look at it. It also welds early tests to the first design you chose, so a big directional change means discarding tests as well as code, and there is a real tendency to over-specify at the unit level, producing exactly the implementation-coupled suites module 3 dissects. Anyone who tells you the costs are zero is selling something.
The honest resolution is per task. Test-first where the behavior is specifiable up front: a rounding rule, a parser, a state machine, a bug fix (write the failing test that reproduces it — this is the highest-value TDD case and nearly uncontested). Test-promptly where it isn't: spike freely, then, before merging, write the tests for the behavior you settled on, treating the spike as disposable. Test-never is not on the menu, and neither is test-first-or-you're-not-serious.
Seven modules of technique, and now the boundary. There is a limit on what any test suite can establish, it was stated precisely in 1969, and it is not a limitation of current tooling that better tooling will lift. It follows from what testing is.
That limit has become sharply more practical in the last few years. When a model writes the implementation and then writes its tests, the suite can be green because the two artifacts agree with each other rather than because either agrees with what you wanted — the oldest asymmetry in the field meeting the newest workflow. This module states the limit correctly, dissects the correlated-blind-spot problem, gives you a protocol for verifying the verifier, and closes by consolidating every anti-pattern in the course into a table you can recognize things by.
Dijkstra: program testing can be used to show the presence of bugs, but never to show their absence. Read it as a statement about the direction of proof, because that is what it is.
A red test is proof. It is a constructive demonstration: here is an input, here is the observed output, here is the expectation it violates. One red test settles the question, and no amount of green anywhere else in the suite touches it.
A green suite is not the mirror image of that. It is the absence of contrary evidence on the claims that happen to be filed. The space of possible inputs and states is effectively infinite; your tests are a finite set of probes; and the fraction of the space they touch is not merely small but unmeasurable — you cannot even quantify what you covered, because coverage counts lines rather than the space (module 6). Green is consistent with a perfect program and equally consistent with a program broken everywhere you did not look.
The correct response is neither reverence nor nihilism. Testing's entire value lies in the direction that does work: finding bugs, cheaply and early, and pinning fixed ones so they cannot return unannounced. What the limit forbids is a specific inference — from green to safe — and what it demands in exchange is that you know what your suite actually claims, which is the skill this whole course has been building.
This is the evidentiary structure you already reason in. A finding of no contrary evidence is not a finding of fact; it is a statement about the record as developed, and its weight depends entirely on how the record was developed — who was deposed, what was requested, which questions were asked. Opposing counsel's whole craft is probing precisely the questions your side did not ask. Read a green suite the way you would read a discovery record produced by the other side: not "nothing was found" but "what was looked for, by whom, and what would I ask that they didn't?"
Now the modern form of the problem. A model writes value_cellar from a specification, then writes its tests from the same specification. Both artifacts pass. What has green measured?
Internal consistency. The tests agree with the code, which is a genuine fact and a much weaker one than it looks, because both were produced by one reading of the spec. If that reading was wrong — if the spec said prices are quoted per case and the model built everything per bottle — then the implementation is wrong, the tests encode the same wrongness as their expected values, and the suite is green with total confidence. This is the verification gap: the blind spots of the code and the blind spots of its tests are correlated, and testing's whole power has always depended on them being independent.
Say precisely what changed and what didn't, because this is not a new phenomenon. A human writing both code and tests in one sitting has correlated blind spots too — that is why pair review and independent QA existed long before language models. What changed is scale and speed: correlated artifacts are now generated in seconds, at volume, and they look excellent. The tests are well-named, well-structured, three-phase, plausibly thorough. Every surface signal a reviewer uses to judge test quality at a glance has been decoupled from the one thing that matters, which is whether the tests encode the intended behavior or the implemented behavior.
There is a particularly sharp version worth naming: tests generated from the code rather than from the spec — "write tests for this function" — cannot fail on any behavior the function currently has. They are the snapshot approve-all reflex (module 7) at the level of an entire suite. They will catch future regressions, which is real value, and they will never once tell you the function was wrong to begin with.
Nº 08 owns the full verification stance for AI-assisted work: how to read generated code, where to demand explanation, how to structure review when volume outpaces attention. This module owns only the test-suite consequence — that a green suite over correlated artifacts measures agreement rather than correctness, and that the human's leverage point is the spec-to-tests joint. If you want the general practice, that guide has it; what follows here is just the cheap mechanical check that tells you whether your suite can fail at all.
The protocol has three parts, and they are cheap enough that there is no excuse for skipping them.
Own the specification, and read the tests as claims first. Reading order matters more than most review advice admits. Read the spec, then the tests, then the code — because reading code first anchors you to what it does, after which the tests read as confirmation rather than as claims to interrogate. Reading the tests against the spec, before you have seen the implementation, is the one position from which a shared misreading is visible. For each test ask module 1's question: what does this claim, and is that a claim I want filed? For the whole set ask the portfolio question from module 1: which risk regions have no claim on file?
Read adversarially. The productive prompt to yourself is: what input would embarrass this suite? You are looking for the unasked questions — the empty cellar, the wine with two vintages, the price of zero, the request that arrives twice. This is where a litigator's instincts are worth more than a testing framework, and it is exactly the reasoning generated tests are worst at, because a plausible-looking suite is generated from the same distribution of expectations as the code.
Sabotage the code and expect red. The mechanical check, and the highest value-per-minute action in this entire course. Mutation testing is the industrial version — tools that systematically mutate operators and report which mutants survive — but the hand version is enough and takes ninety seconds: pick the load-bearing line, break it deliberately, run the tests, and put it back. Change the divisor from 12 to 11. Delete the vintage condition from the join. Flip a comparison. If the suite stays green, you have learned something definite: the tests do not test that behavior, whatever their names say.
Interpret both outcomes carefully. Red is a pass — the suite noticed, and you now have evidence its claims are wired to the code rather than merely adjacent to it. Green is a finding about the suite, not about the code. This is the interpretation people get backwards: they sabotage, see green, and conclude the mutation was harmless. It means the opposite. It means that if the code had ever contained that bug, nothing would have told you — which is the entire question you were asking.
Comment out AND p.vintage = l.vintage in the price join. Run the suite. If it stays green, your integration test either doesn't exist or its fixture has only one vintage per wine (module 4's warning about a fixture that makes the bug unobservable). Change // 12 to // 11. If it stays green, the valuation assertions are vacuous — you have module 6's is not None problem somewhere. Two mutations, two definite answers about what your suite is actually wired to.
The course's anti-patterns, consolidated for recognition. Read the middle column — symptoms are what you actually encounter; the names come later.
| Anti-pattern | Symptom in the wild | Corrective | Module |
|---|---|---|---|
| Reading green as "it works" | A launch review where a dashboard is offered as the answer to "is it correct?" | Restate what green claims; ask which risk regions have no claim filed | m1 |
| Testing the framework | Tests asserting the ORM saved a row or the router routed | Assert your logic at the seam; let the library's maintainers test the library | m1 |
| The ice-cream cone | Forty-minute suite, rotating cast of "flaky" e2e tests, nobody runs tests locally | Create seams, push claims down a level, retire e2e tests whose claims are now covered cheaply | m2 |
| Over-mocking into implementation assertions | Walls of assert_called_with; refactors turn the suite red while incidents keep arriving | For each mock ask what behavior would be wrong if it failed; replace with a fake and assert outcomes | m3 |
| Mocking a boundary you own | A doubled repository or service layer; a behavior-preserving refactor breaks nine tests | Double on the ownership line only; keep owned components real or contract-checked | m3 |
| Mocked-repo tests read as database coverage | "The repository is covered" while no test has executed the query | One integration test against a real disposable database with a fixture that makes the bug observable | m4 |
| Retry-until-green | retries: 3, a known-flaky document, reruns before reads | Root-cause from the four families; quarantine within a day with an owner; track quarantine size | m5 |
| Order-dependent and shared-state tests | Passes alone, fails in parallel; a class-scoped fixture "for speed" | Per-test data inside a rolled-back transaction; run shuffled deliberately | m5 |
| Chasing a coverage number | Rising percentage, is not None assertions, pragmas on the hard files | Drop the hard gate; read a diff's uncovered lines and ask what claim is missing | m6 |
| The tautological property | A property test that transcribes the implementation | Assert a relation checkable by a route the code doesn't take | m7 |
| Approve-all snapshots | Forty diffs re-approved in one keystroke; PR described as "update snapshots" | Snapshots reviewed as code; normalize volatile fields; no bulk update in CI | m7 |
| Trusting AI tests over AI code | Generated code and generated tests, both green, approved on that basis | Read spec → tests → code; sabotage a load-bearing line and expect red | m8 |
| Skipping or deleting the failing test | @skip("flaky, see JIRA-1482") on a test that is not flaky | Only defensible if the claim is re-covered deterministically; otherwise it is unfiling a claim to fake a verdict | m8 |
The last row deserves its own sentence, because it is the course's thesis in its most compressed form. Skipping a legitimately failing test does not make the software work. It removes a claim from the portfolio so that the remaining claims can all be green — which is, precisely and literally, manufacturing a verdict by withdrawing the question. Every anti-pattern above is a version of the same move: spending the confidence budget on bets that produce green without producing information.
You can't test your way to certainty; the limit forbids it. What you can do is know exactly what your suite claims, keep those claims wired to the code, spend the budget where the risk actually is, and refuse the inference from green to safe. That is not a smaller ambition than certainty. It is the only one available, and almost nobody does it.
This is the complete text of the course. With JavaScript enabled, this same page runs the interactive edition — a self-diagnostic that reorders the syllabus around your gaps, knowledge checks, applied worksheets with model answers, and a spaced-repetition review queue — with progress saved locally in your browser.