What's Worth Building

Ask a team where their last year went wrong and you will hear about bugs, a bad migration, an estimate that slipped by six weeks, a dependency that broke in production. These are the failures that generate artifacts: incident reports, red builds, a retro with action items. They are visible because the tooling that detects them is the tooling the industry has spent decades building.

Now ask a different question: of everything the team shipped last year, how much changed anyone's behavior? On most teams nobody can answer, because nothing was set up to answer it. The dominant failure mode in commercial software is not the feature built badly — it is the feature built beautifully, shipped on time, code-reviewed, load-tested, and used by almost nobody. Product judgment is the skill of avoiding that outcome, and it is distinct from every skill your engineering practice is optimized to develop.

The asymmetry in cost is what makes this the expensive failure rather than merely the embarrassing one. A feature built badly for a real need produces complaints, and complaints are a signal — the team iterates, and the second version is better. A feature built well for a need that does not exist produces silence. Silence is indistinguishable from success on a delivery dashboard, so the team ships the sequel. The cost is not the six weeks spent; it is the six weeks plus the maintenance tail plus the opportunity — the bet that was never made because this one occupied the quarter.

The two halves of product work have diverged in difficulty, and the divergence is structural rather than cultural. Building has a feedback loop measured in seconds: the compiler rejects you, the test fails, the reviewer comments, the canary alarms. Every one of those loops has been tightened over decades, and each tightening compounds — better languages make better libraries make better frameworks make better platforms.

Deciding what to build has a feedback loop measured in quarters, and the feedback that arrives is confounded by everything else that happened in those quarters. You shipped a feature; usage rose 4%; so did the marketing spend, and the season changed, and a competitor had an outage. Nothing about that loop tightens on its own. There is no compiler for a bad hypothesis. The only thing that improves it is deliberate practice against written predictions, which is what the last module of this course installs.

This is not an argument that execution is easy. It is an argument that execution is tractable — that effort spent on it reliably converts into competence, and that the industry has built the machinery to make that conversion efficient. Judgment converts effort into competence far less reliably, which is exactly why it stays scarce while build capacity keeps getting cheaper.

For most of software's history, the constraint on shipping value was build capacity. Product managers wrote more good ideas than engineering could implement, and the discipline that mattered was queue management: sequencing a backlog that would never be finished. In that regime, mediocre judgment was survivable, because the build queue absorbed the error. A weak idea sat in the backlog for two quarters and quietly expired.

That constraint is dissolving. Frameworks removed the plumbing, cloud removed the provisioning, and capable models now remove much of the implementation itself: a competent engineer with a good specification produces in a week what took a team a quarter. The queue in front of build shortens. The queue in front of what should exist does not, because nothing in that pipeline got faster — customers still take a week to schedule an interview, behavior still takes a month to show up in the data, and the quality of a hypothesis is still bounded by the person writing it.

The common misreading is that cheap building makes product judgment less necessary — if a feature costs a week, just build it and see. That reasoning survives exactly as long as your build queue is the only cost. It ignores the maintenance tail, the surface area you now have to secure and support, the conceptual weight every added feature puts on the interface, and the fact that a week of build still consumes the team's attention for a week. Cheap experiments are a genuine gift; cheap commitments are a trap, because shipping is a one-way door that costs nothing to walk through and a great deal to walk back.

Discovery, prioritization, and metrics are usually taught as three subjects with three literatures. Treat them instead as one discipline with one job: uncertainty reduction — knowing more, sooner, and more cheaply about whether a bet is right. Discovery reduces uncertainty before you build. Prioritization allocates finite attention across bets whose uncertainty you have now partially priced. Metrics resolve the remaining uncertainty after you build, and honestly enough that the answer can be no.

Two words do a lot of work for the rest of this course, so fix them now. Output is the thing shipped — a cost you incurred. Outcome is the change in behavior it was supposed to cause — the thing you actually wanted. Teams substitute the first for the second because output is countable on Friday and outcome is not. Every anti-pattern in this course is some version of that substitution.

One more framing worth carrying: the cost of learning that an idea is bad rises steeply with how far the idea has traveled. A conversation kills it for free; a launch kills it for a quarter. Module 4 draws that curve and makes it the operating principle of the whole discipline. For now, hold the one-sentence form — learn it left of the build line — and notice how many meetings you sit in where nobody is trying to.

The advice is a cliché and the mechanism behind it is not. Problems are durable: collectors have wanted to remember which bottle surprised them since long before there were apps, and they will want it after yours is gone. Solutions are interchangeable and short-lived — a given implementation is one draw from a large space of things that could serve the same job, and it is almost never the best draw, because you made it before you understood the job.

The practical consequence is about hearing, not sentiment. When a team's identity attaches to a solution, disconfirming evidence arrives as an attack on the team, and people do not evaluate attacks — they defend against them. You have watched this happen: the usage numbers are weak, and the meeting becomes a search for reasons the numbers are wrong. Discovery data gets re-segmented until it agrees. The launch gets a second phase. Every one of those moves is intelligent, and all of them are downstream of a prior commitment that made the honest read unthinkable.

Solution love has a reliable tell: the team can describe the feature in detail and the user's situation only in generalities. Ask "who specifically, doing what, when this happens?" and watch whether the answer sharpens or softens. If it softens, the team is attached to a thing they are building, not a problem they are solving.

A feature idea in its natural state is unfalsifiable. "Users will love sharing" cannot be wrong: love is unmeasured, users is unscoped, and sharing is a category. Nothing you could observe would contradict it, which means no evidence can improve it and no test can end the argument. The first act of discipline is to restate the idea as a falsifiable hypothesis — a claim specific enough that a plausible observation would prove it false.

A workable hypothesis names four things: who (a scoped population, not "users"), what behavior (an action they will take that they do not take now), what magnitude (enough to matter, stated as a number), and by when. The magnitude clause is the one teams skip and the one that does all the work, because without it any nonzero result counts as confirmation.

Notice what the rewrites cost: each one can now fail publicly. That cost is the point. An idea that cannot lose cannot teach you anything, and a roadmap of claims that cannot lose is a roadmap you will defend rather than learn from — a theme module 5 returns to when the roadmap becomes a set of bets.

Here is the wine-cellar team's Tuesday request, verbatim: "We should add social sharing. Two customers asked on calls this week and one emailed. Instagram-style — let people post their bottles."

Three distinct things are fused in that sentence, and the whole skill is pulling them apart. There is an observation (three people raised something in a week — real signal, unknown meaning). There is a job inference, unstated (something about other people is involved in how collectors use this product). And there is a solution, fully specified and stated as if it were the observation (Instagram-style public posting). The solution arrived first because it is the easiest part to imagine, and it is the part with the least evidence behind it.

Requests deserve neither obedience nor dismissal. The customer is an expert witness on their own situation and a novice on your solution space — they know what is painful, when it happens, and what they tried instead; they do not know what is cheap to build, what else you could do, or what the other 41,000 users need. Treat the situation as testimony and the proposed remedy as argument.

The mechanism is two questions asked repeatedly. Upward: why do you want that — what would it let you do? Each answer moves toward the job. Downward, from the job: what else could serve this? Each answer widens the solution space the request had collapsed. A solution that survives the ladder — that still looks best after you have seen its siblings — has earned its place. Most dissolve, which is the point: request laddering is cheap and its failure mode is discovering you were about to build the fourth-best option.

Stop laddering up when the next "why" produces a truism. "Because I want to enjoy wine more" is true, useless, and would justify anything; the last rung with teeth is the one that names a situation, a moment, and a stake. Stop laddering down when the options are no longer materially different — three variations on a share button are one option, not three.

The final move is to put the surviving candidate back on trial. Your reframe is not a conclusion; it is a better hypothesis, produced by the same brain that would have accepted the original request if you had been in a hurry. The team's reshaped claim — collectors will request picks from trusted friends before buying — is exactly as untested as "users will love sharing." It is just cheaper to test and more specific about how it could fail.

The frame is simple and the discipline is in the details: people hire a product, in a specific context, to make progress they care about. The unit of discovery is therefore not a persona and not a feature — it is a job to be done, written as context, motivation, and desired outcome. "Wine collectors aged 35–55 with disposable income" is a demographic; it predicts nothing. "When a bottle surprises me at a restaurant, I want to capture it before I forget the producer, so my next purchase isn't a guess" is a job, and it predicts a great deal — including that whatever you build must work in ninety seconds, in bad light, one-handed, possibly slightly drunk.

A good job statement is falsifiable in the same way a hypothesis is: you can go and find whether people are in that context, whether they take that motivation seriously, and what they currently do instead. That last part — the existing workaround — is the most valuable field in the whole statement, because a job with an ugly workaround is a job someone is already paying for in effort. A job with no workaround at all is usually not a job; it is a preference.

Three question types produce noise reliably. Future hypotheticals — "would you use a feature that…" — ask a person to simulate their own future behavior, which humans do badly and optimistically, and in the presence of the person who built the thing. Leading questions — "how frustrating is it when you can't share a bottle?" — contain their preferred answer; a leading question cannot disconfirm anything, because the only way to answer it is to accept its premise. Compliment solicitations — "what do you think of this design?" — invite social performance, and interviewees are polite.

One question type produces evidence: past behavior, specifically dated. "Tell me about the last time you bought a bottle you weren't sure about — walk me through what you actually did." This works because it asks for recall of events rather than prediction of intentions, because it can surprise you (the answer might be "I never buy bottles I'm unsure about," which is a finding), and because it produces specifics you can check against behavior in your own data.

The say-do gap is not people lying. It is a structural property of self-report: what someone says is data about their self-image and their model of what you want to hear; what someone does is data about the job. The gap widens in exactly the conditions product teams create — when the behavior is socially flattering ("I'd review my portfolio quarterly"), when the cost of the behavior is invisible at question time (nobody imagines the friction of the fourth step), and when the person is being asked by someone who obviously hopes for a yes.

The practical rule is not "never ask." It is: use stated intent to generate hypotheses and revealed behavior to test them. A user telling you they'd love recommendations tells you the topic is live; it does not tell you they will send one. Ask what they did, not what they would do, and where you must ask about the future, price the answer as a lead rather than a finding.

Order candidate tests by evidence per dollar per day, then apply one filter that eliminates most of them: could a plausible result of this test change what we do? If every outcome leads to the same next action, you are not running a test, you are running a ceremony. A discriminating test is one whose plausible outcomes point to different decisions.

This filter is stricter than it sounds. "Show the mockup to ten users and see if they like it" fails, because there is no result that stops the project — enthusiasm ships it and lukewarm feedback triggers a redesign. "Put a fake entry point in front of 12,000 users and ship only if click-through exceeds 5% and at least 40% of clickers complete the waitlist step" passes, because 3% ends the project. The difference is not rigor; it is that one of them has a losing condition.

Notice the third row. A clickable prototype is a fine instrument aimed at the wrong uncertainty most of the time: it answers "can they use it" when the open question is "do they want it." Matching the instrument to the named uncertainty is most of the skill, and module 4 formalizes it as the fidelity ladder.

The most sophisticated failure in this module is the team that does everything above and learns nothing, because the decision was made before the research started. Discovery theater is research performed to confirm a conclusion already reached, and it is not usually cynical — it is what happens when a commitment exists and the research is scheduled to satisfy a process gate.

The symptoms are specific enough to check for. The interview guide is written after the roadmap is published. The team demos the design partway through the session, which converts every remaining answer into feedback on a thing rather than testimony about a life. The stated goal is "validation," a word that only points one direction. Nobody can answer "what result would stop this?" — and when pressed, the answers are unfalsifiable ones. And the giveaway: the sample is drawn entirely from users who already like the product enough to answer your email.

That last corrective is the uncomfortable one and the honest one. A team that has genuinely committed — for strategic reasons, contractual reasons, or because the executive has decided — is better off admitting it than laundering the decision through research. The theater costs real weeks and, worse, teaches the organization that discovery is a formality. Save the instrument for decisions it can actually move.

The wine-cellar team's discovery, run properly: eight past-behavior interviews across two weeks, recruited to include four users who had lapsed. Findings — every one of the eight had a workaround (six texted a specific person, two posted in a group chat); the trusted graph was consistently two to four people, never an audience; and the moment of need was before purchase, in a shop or on a wine list, not after opening. That last finding is the one that reorients the whole bet, and no amount of asking about "sharing" would have produced it.

The same lesson — nobody wants this — has wildly different prices depending on when you buy it. In discovery it costs a conversation. In design it costs a week of someone's work and a little pride. After the build it costs the build, and after launch it costs the build plus migration, support, deprecation, the users you trained to expect it, and the quarter you did not spend on something else.

What makes the curve steep is not just accumulated spend. It is accumulated commitment: public statements, roadmap slides, a team's identity, a customer promise made in a renewal conversation. Sunk cost is a fallacy in economics and a fact in organizations — past a certain point, the psychological and political cost of stopping exceeds the remaining cost of continuing, and teams finish things they know are wrong.

The instruments form a ladder of increasing fidelity and cost, and each rung answers a different question. Choosing the wrong rung is the most common waste in discovery — not because the test is bad, but because it is aimed at an uncertainty you did not have.

Clickable prototype. Two to three days. Answers: can a person accomplish the task, and where do they stall? Cannot answer: will anyone come. Concierge test. One to two weeks of human effort, no code. You deliver the value manually to a handful of users who know it is manual. Answers: does the delivered value actually satisfy the job? This is the strongest instrument for value uncertainty and the most underused. Wizard-of-Oz test. A real-seeming interface with humans behind the curtain. Answers: does the experience work when it feels automatic? Use when the interaction pattern, not the value, is what you doubt. Fake door. A visible entry point to something unbuilt, measuring who acts. Answers: do people want this enough to click, at what rate, in real context? Cannot answer: will they keep using it.

Two practices apply at every rung. First, name the single uncertainty before choosing the instrument — write it as a sentence starting "we don't know whether…" and check that the test discriminates on exactly that. Second, commit the threshold in writing before the data arrives. The number matters less than the order: a threshold set afterward is not a threshold, it is a narration, and every human being on earth will set it just below whatever they observed.

The wine-cellar team's named uncertainty after discovery: we don't know whether enough collectors will act on a pre-purchase request to a trusted friend to justify building a two-sided flow. That is demand uncertainty about an unbuilt behavior, which puts the fake door at the right rung. Cost: four days of one engineer and one designer, no production code touched.

The design. A card appeared in the app's browse and search surfaces reading "Ask a friend — get a pick from someone whose palate you trust." Clicking it opened a screen that said, plainly: we're building this. It doesn't exist yet — we're measuring whether it's worth building. Tell us who you'd ask and we'll let you in first. The screen then asked for one thing: the first name of the person they would ask, and an email for early access.

Thresholds, committed in a doc dated before launch and circulated to the founder: ship-worthy requires (a) click-through of at least 5% of exposed weekly actives and (b) at least 40% of clickers completing the follow-up step. Below either, the bet dies or is reshaped; nothing gets built on a partial pass.

That is a reshape, not a rescue. The original bet — a two-sided request flow — failed its pre-committed threshold and died, on schedule, having cost four days. What replaced it is a different and cheaper thing: a passive "trusted palates" view showing recent bottles from the two to four people you have designated, requiring no request, no reciprocity, and no notification. It is also a new hypothesis with no evidence behind it yet, and the team wrote it down as such.

The arithmetic is not close. A kill in discovery returns the entire remaining build to the team as capacity, and it does so with the option value intact — the team can now spend those weeks on the next candidate, which has not yet been disconfirmed. Reported honestly, the wine-cellar smoke test produced this: four days spent, six weeks of build avoided, one better hypothesis generated. That is a net gain of over five weeks, and it is invisible to every metric the team reports.

Which is the real problem. Opportunity assessment — the structured case for whether a problem is worth pursuing at all — is easy to teach and hard to practice, because organizations count ships. The person who kills a bad idea has produced an absence, and absences do not appear in demos. Three concrete correctives:

Count kills as output. Put them in the same document as the launches, with the avoided cost stated in weeks. "Killed: ask-a-friend request flow. Cost of learning: 4 days. Build avoided: 6 weeks. Replacement hypothesis generated." A team that reports this quarterly changes what its leadership thinks the job is.

Report the counterfactual, not the disappointment. "We didn't build it" invites the question "so what did you do?" "We spent four days to avoid six weeks on something 41,000 users would not have used, and found a cheaper thing they will" answers it in advance.

Make the backlog shrink sometimes. A backlog that only grows is a symptom, not a workload: it means nothing is ever formally decided against, so every stale idea retains the right to be re-litigated by whoever remembers it. Kill items explicitly, with a date and a reason, and keep them in a separate list. The shadow roadmap of things you declined and why is the most useful document a product team owns and the one almost nobody writes.

RICE is reach × impact × confidence ÷ effort. Reach: how many users encounter this in a given period. Impact: how much it moves the thing you care about per user, usually on a coarse scale (3 = massive, 2 = high, 1 = medium, 0.5 = low, 0.25 = minimal). Confidence: your credence in the other three numbers, as a percentage. Effort: person-months.

Its value is not the output. It is that four people who disagree about a priority discover, within ten minutes, which factor they disagree about — and that is almost always the entire argument. Two people arguing about whether to build recommendations are usually not disagreeing about recommendations; one thinks reach is 40% of users and the other thinks it is 6%, or one holds 80% confidence where the other holds 30%. Without the decomposition they will trade conclusions for an hour. With it, they discover in five minutes that the disagreement is empirical and can be resolved with two days of data.

Every framework in this family — RICE, ICE, weighted scoring, value-versus-effort — has the same shape and the same virtue: it forces the factors apart. Pick whichever your team will actually fill in honestly, and change it if people start gaming it, which they will the moment it becomes a target rather than a tool. That failure has a name, and module 7 gives it one.

The four factors have wildly different epistemic status, and treating them as equally solid is what turns a useful tool into false precision.

The consequence is arithmetic: multiplying two well-measured numbers by two invented ones produces an invented number, and reporting it to two decimal places is a claim about precision your inputs cannot support. A RICE of 4.73 versus 4.61 is a tie. Treating it as a decision is not rigor; it is rigor's costume, and the costume is more dangerous than no framework at all because it is harder to argue with.

Two habits keep it honest. Report scores in bands — high, medium, low — or round hard, so nobody can mistake ordering noise for a result. And write confidence as a sentence rather than a number wherever you can: "we have eight interviews and a passed smoke test" is more usable than "80%," because the next person can evaluate the basis rather than the digit.

Value scores answer "how much is this worth?" and are silent on "what does waiting cost?" Those are different questions and the second one determines sequence. Cost of delay is the value lost per unit time that a bet is not delivered, and it separates cleanly into three profiles that sequence very differently even at identical value.

Steady. The bet is worth roughly the same whenever it lands — a recurring efficiency gain, a conversion improvement in a stable funnel. Delay costs a constant rate. Most product work is here, which is why teams forget the other two exist.

Deadline cliff. Value is constant up to a date and collapses after it. Regulatory work is the pure case: a control that must be operating by the enforcement date is worth its full value on the day before and a fraction of it — plus penalties, findings, and remediation cost — on the day after. Delay costs nothing for months and then costs everything at once.

Decaying. Value falls continuously from now: a competitive response, a seasonal feature, an integration with a partner who is also talking to your competitor. Half of the value is gone by the time you get to it in Q3, and no amount of quality recovers it.

Every yes on a roadmap is a set of nos you did not write down. The visible roadmap has four items; the real one has those four plus the thirty you declined, and the second list is where all the information is. It tells you what the team believes, what it has ruled out and on what evidence, and whether anyone can tell you why. Keep it — dated, with reasons — as the shadow roadmap. It ends the re-litigation of settled questions and it is the single best onboarding document a new team member can read.

Two failure modes dominate the deciding conversation itself.

The HiPPO. The highest-paid person's opinion functions as a prioritization framework in most organizations, and the two obvious responses are both wrong. Silent capitulation destroys the team's ability to reason — everyone learns the factors are decorative and stops filling them in honestly. Open veto costs you the political capital you need for the next twelve decisions and usually loses anyway. The corrective is to route the demand through exactly the same structure as everything else: "Great — let's score it. What's your read on reach?" This is not a delaying tactic. Executives frequently have real information the team lacks: a deal that closes on it, a competitive move, a board commitment. Routing through the factors surfaces that information if it exists, and surfaces its absence if it does not, without anyone having to say the word "no" out loud.

Loudest voice. The same pathology at peer level: priority accrues to whoever is most persistent, most senior in tenure, or most recently in a customer meeting. The corrective is identical — the argument happens over factors, not volume — with one addition: write who claimed what, so that when reach turns out to be 6% rather than 40%, the calibration is attributable. That is not for blame. It is the only way anyone's forecasting improves, which module 8 develops into the decision journal.

A roadmap item written as "ship trusted palates v1" has a property worth noticing: it cannot be wrong. It can be late, and it can be descoped, but the day it ships it is unambiguously achieved regardless of whether anything improved. That is what makes it comfortable, and it is the same property that makes it useless as a decision record.

A bet is the alternative: an outcome, a magnitude, and a confidence, stated so it can lose. "Raise the share of new cellars reaching 10 logged bottles in 7 days from 29% to 36% by end of Q3; medium confidence — one passed smoke test, no comparable prior launch." This item can fail while shipping on time, which is precisely the information the output framing suppresses.

The shift has a consequence teams resist: it means the feature is a cost, not an achievement. You spent six weeks to chase an outcome; the six weeks are the price, the outcome is the purchase, and shipping is merely the moment you find out whether you bought anything. A team that internalizes this will kill its own launched features, which is the behavior module 8 requires.

Dates on a roadmap communicate two things simultaneously — sequence and commitment — and they are almost always wrong about the second while being unnecessary for the first. A now/next/later board separates them: now is in flight and its scope is known, next is the shortlist whose evidence is gathered but whose sequence may shift, later is directional and explicitly unpromised. Confidence decays with horizon, which is honest, and the format makes that decay visible instead of hiding it behind three equally crisp Gantt bars.

Commit to a real date exactly where coordination genuinely requires one: a regulatory enforcement date, a contractual commitment, a partner launch, a conference, a dependency another team is sequencing against. Those dates are real and should be defended. Every other date is a forecast presented as a promise, and it converts the team from learners into defenders — because once the date is the commitment, the honest mid-course finding "this isn't working" costs more to say than it costs to ignore.

The structure for that answer, generalized: what is committed (with a date, if a date is real), what is conditional (with the condition named), and what would change it (a specific, observable event). Three sentences. It is honest, it is plannable, and it does not require you to know the future — only to be precise about which parts of it you do not know.

The definition that makes everything else follow: a metric is useful exactly to the extent that some value of it would change what you do next. Apply that as a test, mechanically. Take a number on your dashboard, imagine it halving, and ask what would happen. If the answer is "someone would mention it in standup," you have decoration. If the answer is "we would stop the rollout and go look at the onboarding funnel," you have an instrument.

This reframing is more demanding than it looks, because it disqualifies a lot of numbers people are fond of. Total registered users cannot halve. Cumulative anything cannot halve. Numbers with no owner cannot trigger an action because nobody is accountable for taking one. And numbers with no threshold cannot trigger an action because there is no defined value at which anyone is supposed to do anything — which is why the threshold, not the metric, is where the decision actually lives.

A North Star metric is the single number a team chooses to proxy delivered value — the thing that, if it rises honestly, means users got more of what they came for. It has three properties: it is value-linked (it cannot rise while users get less), movable in aggregate (the team's work plausibly affects it within a horizon they can observe), and it leads revenue rather than trailing it (it changes before the money does, so it can steer).

The wine-cellar team's choice: weekly active cellars with at least one update — currently 3,410. Here is what they rejected and why, because the rejections teach more than the choice.

Note what the winning definition smuggles in: "at least one update" excludes passive opens, which matters because a collector who opens the app and closes it has received nothing. Definitions do enormous work in metric design, and most metric disputes are definition disputes wearing a disguise.

Two orthogonal cuts, frequently confused. Input versus output is about controllability: an input metric is something a named team can directly move this quarter through work they can plan; an output is a result those inputs sum into. Leading metric versus lagging metric is about timing: leading signals arrive while there is still time to act, lagging ones confirm after the fact.

A metric can be any combination. Revenue is a lagging output. Weekly active cellars is a leading output — it moves before revenue and is still not something anyone can move directly. New-cellar activation rate is a leading input: a named team can change onboarding this quarter and watch it respond within weeks. Support ticket volume is a lagging input.

Teams should be accountable for inputs and evaluated on outputs. Holding a team accountable for an output it cannot move produces either learned helplessness or creative accounting, and module 7 covers the second one. Steering by leading inputs is not a preference; it is the only way to act while acting still helps.

The instrumentation contract is a short document written before the build, containing five things: the events you will emit, the metric definitions computed from them, the success threshold, the read date, and the decision rule — what happens at each side of the threshold. It is signed, in the ordinary sense that people have read it and agreed, before any code exists.

Its purpose is not tidiness. It is that after launch, everyone involved will have an opinion about what success looks like, and those opinions will be shaped by the result. The contract removes the most consequential decision — what counts as working — from the period when the answer is known. This is the same mechanism as the smoke-test threshold in module 4, applied one stage later, and module 8 names it pre-registration.

The opposite failure of a vanity metric is an aspirational one: a metric so lagging and so aggregate that the team responsible for it has no lever. A feature team told to move "company revenue" or "overall retention" is being handed a number they can affect only through a chain of six causes, most owned by other people, over a horizon longer than their planning cycle. The predictable results are learned helplessness (the metric is reported and ignored) or narrative (quarterly stories about how the work contributed, unfalsifiable by construction).

The corrective is not a humbler North Star. The North Star should be ambitious — that is its job. The corrective is a properly built input layer: decompose the star into three or four metrics that a named team can move this quarter through work they can plan, and hold the team to those while evaluating the star. Decomposition is where most metric programs fail, because it requires an actual causal model of the product, and writing one down means committing to beliefs that can be wrong.

A vanity metric is a number that only goes up and changes no decision. Cumulative signups, total downloads, total shares created, total documents processed since launch — every one of them is a true statement, and every one is unfalsifiable as a claim about health, because there is no value it could take that would mean something is wrong. A product losing users every week posts record cumulative totals until the day it shuts down.

The diagnostic is one question: what would we do differently if this number flattened? If nothing, it is decoration or public relations, not measurement. The reliable tell is monotonicity — anything cumulative, anything "total ever," anything that resets only upward. The second tell is the absence of a denominator: "14,900 shares created" is a vanity metric and "share of weekly actives who shared at least once, and their 4-week retention versus non-sharers" is an instrument, computed from the same events.

Goodhart's law: when a measure becomes a target, it ceases to be a good measure. The mechanism is not cynicism and it does not require dishonest people. A metric is a proxy — a cheap observable correlated with an expensive-to-observe outcome. That correlation held under the behavior that existed when you chose the metric. The moment you attach consequences to the number, you recruit optimization pressure toward the number itself, and optimization finds the cheapest path to it. The cheapest path is almost never the outcome, because the outcome is the expensive thing you were proxying around in the first place.

Three properties predict how badly a metric will Goodhart. Proxy distance: how many steps sit between the number and the outcome. Gaming surface: how much of the number the people measured control directly, including definitions and categorizations. Consequence strength: how much rides on it — bonuses and headcount produce more pressure than a dashboard nobody reads.

A financial-services GRC program, four quarters, reconstructed from its own records.

Q1 — the metric is chosen. The program tracks findings closed per quarter across application security, third-party risk, and access reviews. Baseline: 210. It is a reasonable proxy. Closing findings genuinely was, at that point, correlated with risk going down, because the team closed things by fixing them — that was the only way anyone knew how.

Q2 — the metric becomes a target. An executive commits to the board that findings closed will reach 400 per quarter, and the number enters the leadership scorecard tied to team bonuses. Nothing about the underlying work changes. Proxy distance was already moderate; gaming surface is very large, because this team writes the findings, assigns their severities, and decides what "closed" means; consequence strength just went from a dashboard to a bonus. All three dials are now set to maximum.

Q2–Q4 — the divergence. Reported closures: 210 → 340 → 468 → 512. Every number is accurate. Three behaviors produced them, none of which anyone would describe as fraud:

Splitting. A single finding — "37 EC2 instances missing endpoint agent" — is re-recorded as 37 findings. Same work, same risk, 37 closures. This is genuinely defensible as a tracking practice, which is exactly why it is the first move.

Severity drift. Critical findings take weeks and require change windows; mediums can be closed with a compensating control and a paragraph. Over three quarters, the share of new findings rated critical falls from 18% to 6%, with no change in the underlying environment. Nobody downgraded anything dishonestly; the definitional boundary simply drifted in the direction that made the quarter work.

Close-and-reopen. Findings are closed at the quarter boundary on the strength of a remediation plan, then reopened in the following quarter when the plan does not complete. Each cycle produces a closure. One access-review finding is closed four times across the year.

The countermetrics that were not being watched. Median age of open critical findings: 34 days → 71 days. Recurrence rate within 90 days: 4% → 22%. Findings discovered by external audit rather than internal process: 11% → 29% — the sharpest signal, because it measures what the internal process is now missing. The chart that leadership saw improved by 144% over four quarters while every one of these deteriorated.

The walkback. The moment the measure stopped measuring is not when someone gamed it — it is the day the target was announced with consequences attached. Everything after that was rational behavior by people responding to the incentive they were given, and treating it as an integrity problem both insults them and guarantees a repeat, because the next metric will be gamed the same way by the same rational people.

The countermetric. The fix is to pair every target with a metric that would deteriorate if the target were gamed, and to publish them together so neither can be quoted alone. Here: closures per quarter paired with median age of open criticals (defeats close-and-reopen and severity drift, since both leave old criticals sitting) and 90-day recurrence rate (defeats splitting and premature closure). Choose countermetrics by asking the uncomfortable question directly: if I had to hit this number without doing the work, how would I do it — and what would that damage?

Every feedback channel you own is a filter, and the filter runs before you see anything. In-app surveys reach people currently in the app. NPS reaches people who did not churn. Interviews reach people who answered your email. Support tickets reach people who cared enough to complain rather than leave. Sales calls reach people still in your pipeline. Each channel produces true statements about the population that reached it, and each is silent about the people whose absence is the thing you most need to explain.

Selection bias is the general case: conclusions drawn from a sample your own process filtered. Survivorship bias is its sharpest form — hearing only from those who stayed. The wartime case is worth stating correctly, because it is usually told backwards: analysts examining returning bombers found damage concentrated on wings and fuselage and proposed armoring those areas. Abraham Wald's insight was that the sample was defined by having returned, so the observed damage marked the survivable hits. The armor belonged where the returning planes showed no damage — the engines — because planes hit there did not come back to be measured.

The correctives are cheap and almost nobody runs them: recruit from the churned list rather than the active list, sample from all exposed users rather than all responders, and — the highest-yield habit — write down what population your conclusion is actually about before you state it. "Users love it" is nearly always "the 6% of exposed users who both stayed and answered love it," and once you write it that way, most of the temptation dissolves on its own.

An average is a summary that assumes the thing it summarizes is one thing. When a population contains groups moving in opposite directions, the average reports their weighted cancellation, and a flat line is the most dangerous shape a metric can take — because it looks like stability and is frequently two stories in violent disagreement.

The wine-cellar case: four-week retention holds at 41% for two quarters. Cut by cohort tenure, it is two lines. Users with more than 50 logged bottles rose from 62% to 71%: the product got better at serving established collectors, which is real and worth knowing. Users in their first month fell from 34% to 22%: onboarding degraded badly, plausibly because the features serving the power segment added surface area a new user must now navigate. The aggregate moved 0.4 points. Both facts are large, both are actionable, and the metric that leadership watched reported neither.

The operating rule: cut every metric by the segments the decision affects before you trust it. Which segments those are is itself a judgment call, and the four that pay off most often are tenure or cohort, plan or price tier, platform, and acquisition channel. If the decision is about onboarding, tenure is not optional. If it is about pricing, tier is not optional.

A launch review can do exactly one of two things, and which one is fixed before the review begins. If the success criteria were written down before the data existed, the review evaluates a bet. If they were not, the review narrates a result — and everyone in the room, acting in good faith, will construct a threshold that the observed number happens to clear.

Pre-registration is the practice that separates them: the metric, the threshold, the read date, and the decision rule committed before launch, plus an explicit list of moves that are out of bounds afterward. That list matters as much as the threshold, because the pressure at read time never arrives as "let's change the bar." It arrives as three reasonable-sounding proposals: extend the window because the trend looks promising, switch to a metric that did move, or find the segment where it worked.

The wine-cellar team built the trusted-palates view. It shipped on time, at high quality, and the team liked it — the passive feed of what two to four trusted collectors had recently logged was, by any craft standard, the best thing they had made that year. They also shipped a lightweight share action alongside it, because it was two days of work and the feed made it natural.

Six weeks after full rollout, on the read date written into the contract in May:

The +1.2% sits inside the noise band the contract specified. The bar was +6%, chosen in May against two prior launches that moved 5% and 9%. This is a fail, and it is not a close one dressed up as ambiguity — it is a fail with a large, cheerful, entirely genuine counter next to it reading 14,900.

The decision, executed as written. +1.2% is below the +2% floor, so the contract's bottom branch applied: the feature came off the primary surfaces. The team wrote it up in the shadow roadmap with the numbers and the date. Six weeks of build, killed by its own authors, against a bar they set themselves in May.

The salvage, which is real and is not a rescue. One input metric moved: 310 return visits per week attributable to a palate update. That is small against a 3,410 base and it is not nothing — it says the notification, not the feed, was the part that did work. The next bet, entering the funnel at opinion stage with no validation yet, is a drinking-window notification that names a specific trusted collector who logged the same bottle. It is a different bet with its own contract, and calling it a continuation of this one would be exactly the hindsight rewriting the next section is about.

Judgment evaporates at handoff unless something carries it. The common failure is a spec that describes the artifact perfectly — screens, states, edge cases, acceptance criteria — and says nothing about why it exists or what would make it wrong. That spec produces a correctly built thing that nobody can evaluate, and it strands the builders: an engineer who knows the bet can protect it during the hundred small decisions where the bet actually lives, and an engineer who knows only the screens cannot.

An executable spec should inherit four things from everything upstream of it. The job (context, motivation, outcome, and the workaround it must beat) — this is what lets a builder resolve an ambiguity without a meeting. The hypothesis, with its magnitude, so the scope conversation has a reference point. The metrics and events from the instrumentation contract, specified as build requirements rather than a follow-up ticket. The kill criteria and read date, so that everyone building it knows from day one that it can be removed, and by what rule.

The last item is the one that gets cut, and cutting it is what makes a kill impossible later. A team that learns on the read date that the feature might be removed will fight it. A team that knew in week one, and helped write the criterion, will run the removal themselves.

Memory is reconstructive, and it reconstructs toward coherence. Six weeks after a result, everyone involved sincerely remembers having had reservations, and the version of the bet they remember making is subtly better calibrated than the one they made. This is not dishonesty; it is how memory works, and it makes unaided retrospection nearly worthless for improving judgment.

The instrument is the decision journal: a short written entry at decision time recording the hypothesis, the evidence and its gaps, the predicted outcome with a confidence number, the success threshold, the kill criteria, the read date, and the alternatives declined. It takes fifteen minutes and it is the only admissible record when the post-mortem convenes, because it was written by someone who did not know the answer.

Run the review against that document and nothing else. The two questions that matter: was the process sound given what we knew then? and what would we have needed to know to decide differently, and could we have bought it cheaply? Note that neither question is "did it work."

Taste is not a gift; it is compressed feedback from honestly closed loops. Someone with twenty years of experience and no closed loops has one year of experience twenty times, because nothing ever came back and told them they were wrong in a form they could not reinterpret. Calibration improves under exactly one condition: predictions written down in advance, with confidence attached, and scored later against what happened.

That scoring requires one more separation, and it is the one people find hardest. Resulting is judging a decision by its outcome rather than by the process and information available at the time. Decisions and outcomes are two variables, and all four combinations occur. A well-reasoned bet with good evidence can fail because the world moved. A guess with no evidence can win because it was lucky. If you evaluate only outcomes, you will learn to imitate lucky guesses, which is worse than learning nothing.

The field checklist. Ten anti-patterns, each with a symptom you can observe and a corrective you can run. This is the course, compressed.

One habit carries more of this than any other, and it is the cheapest: before every meaningful bet, write down what you predict, how confident you are, and what would change your mind. Fifteen minutes. Score them quarterly against what happened. That is the entire mechanism by which product judgment stops being a personality trait and starts being a skill — and it is the one thing in this course that nobody can do for you.