The Meter Is Running · Nº 11
A field guide to the unit economics of AI products
Somewhere around 2010 the software industry stopped thinking about the cost of serving one more request. It was a reasonable thing to stop thinking about: hosting a marginal user cost a fraction of a cent, storage fell faster than usage grew, and the entire discipline of SaaS product management calcified around a convenient fact — that once the software existed, running it again was free. Free trials, unlimited plans, engagement as an unqualified good, growth as the answer to any margin question: these are not eternal truths of product strategy. They are local adaptations to a cost structure.
AI features break that cost structure. Every model call spends real money, the spend scales with exactly the design choices product people control, and it compounds in multi-turn and agentic flows. This module establishes the frame the rest of the course assumes: an AI feature's cost of goods has a distinctive anatomy, that anatomy is fixed at design review rather than at the monthly close, and a PM who cannot model cost per action to within a factor of two is not qualified to approve an agent loop.
Marginal cost is the cost of serving one more unit of usage. For a mature SaaS product it is close enough to zero that nobody measures it per action: the servers are already running, the database is already provisioned, and the thousandth page view of the day costs materially the same as the first — which is to say, nothing you would put on a slide. That single fact generated most of the product instincts you inherited. Unlimited free tiers work because free users cost almost nothing to carry. Engagement is unambiguously good because more usage is more retention at no incremental cost. Growth solves margin problems because the fixed costs amortize across a larger base. Each of those is a correct inference from a premise.
An AI feature falsifies the premise. When a support agent asks your product to draft a reply, a model provider meters the tokens and sends you a bill. Nothing amortizes. The thousandth draft of the day costs the same as the first, and the millionth costs the same as the thousandth. If the draft costs you 1.6 cents and the seat that generated it is priced at forty dollars a month, then the fortieth dollar of margin is a function of how many drafts that agent produces — a number set by the customer, not by you, unless you designed a limit into the product.
SaaS margins improve with usage because usage amortizes fixed cost. AI margins degrade with usage because usage is variable cost. Every product reflex you built on the first sentence inverts under the second, and the inversion is silent — the feature ships, the engagement dashboard goes up and to the right, and the damage shows up six weeks later in a line item nobody was watching.
The phrase to retire is "scale will fix it." Scale fixes costs that amortize. A negative unit margin — losing three cents per use — does not improve at a million uses per day; it becomes a thirty-thousand-dollar daily loss with an excellent growth chart on top of it. This is not a subtle error, but it is a common one, because "we'll figure out margins at scale" was true for so long that it stopped sounding like a claim and started sounding like a maxim.
COGS — cost of goods sold — is the set of costs that scale with delivering the product, and it is the denominator of every gross-margin conversation you will have with finance. A traditional SaaS feature's COGS stack is short and mostly amortizing: compute and hosting, storage and bandwidth, third-party services, and a share of support. Volume growth spreads the first three across more revenue; only support scales anything like linearly, and even support is sublinear because documentation and self-service absorb the easy cases.
An AI feature adds four lines with different behavior. Inference is the dominant one and it is purely variable — tokens in, tokens out, priced per million, no amortization anywhere. Retries and regenerations are the tax on non-determinism: the user does not like the first draft, hits regenerate, and you pay again for the same user intent. Evaluation and monitoring is the cost of knowing whether the feature still works — golden-set runs on every deploy, LLM-judge scoring, sampled production grading — and it scales with release cadence and with the number of behaviors you are defending. Human fallback is the cost of the cases the model does not handle: the escalation to a person, the manual review of a flagged output, the compliance read on a high-risk answer. That last line is the one teams forget, and it is the one that makes cheap models expensive.
Teams commonly model only the inference band and call it the unit cost. That understates the real number by anywhere from twenty percent to several multiples, because the regeneration rate, the eval budget, and above all the human-fallback rate are exactly the lines that a cheaper model inflates. A model that is half the price and produces twice the escalations is not a saving; it is a transfer from your infrastructure budget to your support payroll, where it is larger.
Here is the claim the rest of this course elaborates: the variables that determine an AI feature's unit cost are set by product and engineering decisions, not by finance. How much context you retrieve. How long the system prompt is. Whether the prompt prefix is stable enough to cache. Which model tier handles which task. How many turns a conversation runs before the product nudges toward resolution. How many iterations an agent loop is permitted. Every one of those is decided in a design review, months before anyone sees a bill — and every one of them moves the unit cost by a factor, not a percentage.
That timing is what makes "we'll model the cost after launch" so expensive. By launch the retrieval strategy is built, the prompt is tuned, the loop is written, and the customers are using it. Changing the model tier after the fact is a quality regression you now have to defend; changing the context strategy is a rewrite; changing the loop budget is a visible product change. The cheap moment to move a unit cost by 5x is the moment when the design is still a document.
Law firms never lost marginal cost. Every matter consumed hours, every hour had a rate, and the discipline of knowing where each one went was not a finance function — it was how associates were trained to think about the work itself. That instinct, which software product management spent thirty years unlearning because it had no object, now has an object again. Reading a cost model for an agent loop is not meaningfully different from reading a matter budget: stated assumptions, a defensible estimate per step, and an explicit answer to "what happens if this runs long."
The practical form of the discipline is small. For any feature that calls a model, you should be able to state, on demand and without a spreadsheet open: what one action costs, what one workflow costs, what one seat costs per month, and which two assumptions the answer is most sensitive to. If you cannot, then the number is being decided by defaults — the default context window, the default model, the default retry policy — and defaults are chosen by engineers optimizing for the demo, not by anyone optimizing for margin.
Throughout this course, one product carries the arithmetic: Harbor, an AI draft-reply tool for customer-support agents. An agent opens a contact, Harbor retrieves relevant knowledge-base articles and the customer's ticket history, and drafts a reply the agent edits and sends. It is deliberately unglamorous — a single, bounded, high-volume action — because that is the kind of feature whose unit economics are decidable, and the kind most PMs will actually ship. Module 3 costs it end to end.
You cannot model a cost you cannot decompose. This module takes one Harbor request apart into the four things you are actually paying for, prices each at the correct rate, and then does the thing that separates a real cost model from a plausible one: it follows the same request across ten turns of a conversation and watches the bill grow even though the user's messages do not.
Two results here are counterintuitive enough to be worth stating up front. First, output tokens cost several times input tokens, and yet for context-heavy products the input side dominates the bill — so the optimization that feels obvious (make the answers shorter) is usually the wrong one. Second, prompt caching is not a discount; it is a conditional discount whose condition is a measured, decaying property of your prompt design. Teams that treat the cache hit rate as a constant build cost models that are correct on the day they are written and wrong every day after.
A token is the billing quantum of model usage — roughly three-quarters of an English word, though the ratio varies by language and by how much code, punctuation, and JSON is in the text. Everything you send is metered as input tokens; everything the model generates is metered as output tokens. The two are priced differently, and the difference is not arbitrary.
Processing input is parallel work: the model reads the whole prompt at once, and the arithmetic that does it saturates the hardware efficiently. Generating output is serial: each token is produced by a full forward pass conditioned on everything before it, one at a time, and the hardware sits underutilized while it happens. The result is a durable input/output asymmetry — output typically costs roughly five to eight times input on the same model. That ratio is a fact about how transformers run, not a pricing whim, so it is more stable than absolute prices — though it still varies by provider, running about 5x on Claude and 6–8x on the current GPT-5 and Gemini tiers.
Every dollar figure in this course uses one pinned, illustrative rate card so the arithmetic is followable: frontier $3.00 per million input tokens and $15.00 per million output; mid-tier $1.00 / $5.00; small $0.25 / $1.25; cache reads at 10% of the input rate; cache writes at 125% of it; batch at 50% off. These are in the neighborhood of published mid-2026 pricing, and they will be wrong by the time you need them. The canonical source is your provider's published pricing page, checked at the moment you make the decision, with the date of the check recorded in your cost model. A cost model without a verification date on its prices is a rumor with arithmetic attached.
The asymmetry has an immediate design consequence. For a workload that reads a lot and writes a little — classification, extraction, routing, retrieval-augmented drafting — the input side carries the bill despite the higher output rate, simply because there is so much more of it. For a workload that reads a little and writes a lot — long-form generation, code synthesis, translation of short prompts into long artifacts — output dominates. Knowing which shape your workload has tells you which lever to pull first, and it is the first thing to establish about any feature you are asked to cost.
Harbor's draft request carries four distinct payloads, and it is worth being precise about them because they behave differently under every optimization in this course. The system prompt is 2,500 tokens of role definition, tone rules, escalation policy, and output format — identical on every request. The retrieved context is 3,500 tokens of knowledge-base articles and prior-ticket excerpts pulled by the retrieval step; it changes per contact but is assembled from a stable corpus. The dynamic suffix is about 1,000 tokens: the customer's inbound message and the agent's notes. The output is a 400-token draft reply.
Priced at the pinned frontier rates: input is 2,500 + 3,500 + 1,000 = 7,000 tokens at $3.00 per million, which is $0.021. Output is 400 tokens at $15.00 per million, which is $0.006. One uncached Harbor draft costs $0.027.
The practical reading: an engineer who shortens Harbor's drafts from 400 to 200 tokens saves $0.003 per request, or 11 percent. An engineer who tightens retrieval from seven knowledge-base chunks to four — 3,500 tokens down to 2,000 — saves $0.0045, or 17 percent, and probably improves quality by removing distractors. Both are worth doing. Only one of them is where the money is, and the intuition trained on "output is expensive" points at the wrong one.
Models are stateless. A conversation feels continuous to the user because the client re-sends the entire transcript on every turn — and you are billed for all of it, again, every time. This is context accumulation, and it is the single most under-modeled mechanism in AI product economics.
Take a Harbor contact where the agent iterates with the assistant: refine the tone, add a policy caveat, shorten it. Assume each turn adds about 700 tokens of history (the previous draft plus the agent's instruction). The system prompt and retrieved context stay at 6,000 tokens, and the dynamic suffix stays at 1,000. Turn 1 sends 7,000 input tokens and costs $0.027. Turn 5 carries 2,800 tokens of accumulated history, sends 9,800, and costs $0.0354. Turn 10 carries 6,300 tokens of history, sends 13,300, and costs $0.0459 — 1.7 times turn 1, for a request the user perceives as identical work.
The cumulative arithmetic is the part that hurts. Across ten turns, Harbor sends 101,500 input tokens and 4,000 output tokens: $0.3045 plus $0.06, or $0.3645 for the conversation. The naive estimate — turn-1 cost times ten — gives $0.27. The error is 35 percent at ten turns. Run the same conversation to thirty turns and the true cost is $1.72 against a naive estimate of $0.81: the estimate is off by more than half, and the direction of the error is always the same.
Symptom: a cost model whose per-request number was measured from a single-turn test, multiplied by an average session length taken from an analytics dashboard. Why it fails: per-turn cost is not constant; it is a growing function of turn number, so the product of a first-turn cost and a turn count understates by a factor that grows with conversation depth. Corrective: model the sum, not the product — compute cost per turn as a function of accumulated context and add up the series, then sanity-check it against the p50 and p95 observed session lengths rather than the mean.
Two mechanisms change the arithmetic rather than the design, and both are conditional.
Prompt caching lets the provider retain the processed state of a stable prompt prefix and re-price it on subsequent requests — at the pinned rates, 10 percent of the input rate for a read, in exchange for a 25 percent premium the first time the prefix is written and a time-to-live measured in minutes. Harbor's system prompt plus retrieved context form a 6,000-token prefix that is identical across every request in a contact and largely shared across agents working the same product area. Suppose the measured behavior is: 70 percent of requests read a warm cache, 22 percent arrive against a prefix that is expired or being concurrently rewritten and pay the plain input rate, and 8 percent trigger the write. The effective prefix rate is 0.70 × $0.30 + 0.22 × $3.00 + 0.08 × $3.75 = $1.17 per million against a list rate of $3.00. The prefix costs $0.00702 instead of $0.018. Add the 1,000-token dynamic suffix at $0.003 and the 400-token output at $0.006, and a cached Harbor draft costs $0.016 against $0.027 uncached — a 41 percent reduction.
Batch processing is the other machine: submit a set of requests, accept a completion window measured in hours rather than seconds, and pay roughly half. It is not a lesser service, it is a different product — the provider is buying the right to schedule your work into idle capacity, and it pays you for that right in the only currency you have, which is patience. Whether you have any to spend is a property of the workload, not of the model.
Harbor's interactive draft path has none: an agent is looking at a spinner with a customer waiting, and a four-hour completion window is not a degraded experience but the absence of one. Harbor's nightly quality-summary job — which reads every closed contact and produces per-team coaching notes — has nothing but patience, and running it on the interactive path was simply an unexamined default that cost the company half of a five-figure line item.
Symptom: a cost model with "cache hit rate: 95%" typed into a cell, with no adjacent dashboard. Why it fails: the hit rate is a measured property of prompt stability and traffic shape, and it decays. Any edit to the system prompt invalidates every warm prefix at once. A retrieval change that reorders chunks invalidates them per contact. TTL expiry punishes low-traffic tenants and off-hours traffic. A tenant with a custom tone instruction may have a hit rate near zero while the fleet average looks healthy. Corrective: instrument the hit rate as a first-class metric, carry it in the cost model as a range with the low end taken from your worst tenant, and put a deploy-time alert on the drop that follows every prompt edit.
This is the module where the course stops describing mechanisms and starts producing numbers. You will build Harbor's cost model three times over, at increasing scope: what one draft costs, what one customer contact costs, and what one agent seat costs per month. The last number is the one you take into a pricing conversation, and it is the one that is almost never computed before the pricing conversation happens.
Two disciplines make the difference between a model and a guess. The first is stating every assumption inline, next to the arithmetic it feeds, so a reader can attack any single number without reconstructing the whole thing. The second is sensitivity analysis: varying each assumption and ranking the results, so you know which two numbers deserve a dashboard and which five you can safely stop arguing about. An analyst who tells you nine variables matter has not finished the analysis.
A cost per action is four honest numbers multiplied: tokens in, tokens out, the rates, and — if you cache — the effective rather than list input rate. Harbor's pinned assumption table is the input to everything that follows in this course, so it is worth reading once carefully.
| Assumption | Value | Basis |
|---|---|---|
| System prompt | 2,500 tokens | Measured from the shipped prompt file |
| Retrieved context | 3,500 tokens | p50 of four KB chunks plus two ticket excerpts |
| Dynamic suffix | 1,000 tokens | p50 customer message plus agent note |
| Draft output | 400 tokens | p50 of shipped drafts |
| Drafts per contact | 1.4 | Measured: 40% of contacts request one regeneration |
| Contacts per agent-day | 45 | From the support platform, median agent |
| Workdays per month | 21 | Calendar |
| Cache hit rate on the 6,000-token prefix | 70% read / 22% miss / 8% write | Measured over four weeks, fleet-wide |
| Rates (frontier) | $3.00 / $15.00 per M | Illustrative mid-2026; verify at decision time |
Uncached. Input 7,000 tokens × $3.00/M = $0.021. Output 400 × $15.00/M = $0.006. $0.027 per draft.
Cached. The effective prefix rate is 0.70 × $0.30 + 0.22 × $3.00 + 0.08 × $3.75 = $1.17 per million. The 6,000-token prefix therefore costs $0.00702; the 1,000-token dynamic suffix costs $0.003 at the list input rate; output is unchanged at $0.006. $0.01602 per draft — call it $0.016.
Every figure in this module rests on the pinned rate card from Module 2. Those rates are illustrative and will drift. Before any of this arithmetic is used to set a price, sign a contract, or approve a build, re-verify the rates on the provider's published pricing page and record the date of the check in the model itself. The verification date belongs beside the number, not in a changelog nobody opens.
The effective input rate — not the list rate — is the number that belongs in a cost model, and it is a property of your architecture rather than of the provider's price sheet. Harbor pays $1.17 per million for its prefix because of a design decision about prompt stability. Two teams buying from the same vendor at the same list price can be paying rates that differ by 2.5x, and neither one will discover it from the invoice.
An action is not a unit anyone buys. Customers buy seats, or contracts, or outcomes, so the model has to travel up two levels before it means anything commercially.
Action to workflow. One customer contact does not consume one draft; it consumes 1.4, because 40 percent of contacts get a regeneration. This is the step teams skip, and skipping it understates by 40 percent — the happy path is not the path. Cost per contact is 1.4 × $0.01602 = $0.02243.
Workflow to seat. A median agent handles 45 contacts a day across 21 workdays: 945 contacts and 1,323 drafts per month. Cost per seat-month is 1,323 × $0.01602 = $21.19. Without caching the same seat costs 1,323 × $0.027 = $35.72, so the prompt-caching architecture is worth $14.53 per seat per month — on a 500-seat account, $7,265 a month, which is a real number obtained by a design decision rather than a negotiation.
A cost model with nine assumptions invites nine arguments. A sensitivity analysis ends eight of them. The method is mechanical: vary each assumption by ±50 percent, hold the others at baseline, recompute the output, and sort by the size of the swing.
The ranking needs one further cut, because "large swing" and "useful lever" are different properties. Contacts per agent-day is the largest swing and the least actionable: it is set by the customer's ticket volume and staffing, you will never move it, and it is a number to measure per account rather than to model. Drafts per contact is equally large and entirely yours — it moves with draft quality, with whether you cap regenerations, and with how the regenerate button is presented. Cache hit rate is third by swing and first by leverage per unit of engineering effort.
So the defensible statement is: the two variables to model, monitor, and design against are drafts per contact and cache hit rate. Contacts per agent-day is not an assumption at all — it is an account attribute that belongs in the per-customer instantiation of the model, not in its sensitivity table. The remaining four together move cost per seat by roughly $10 combined at ±50 percent — less than the cache alone — which is below the noise in the volume terms, and should be fixed at their measured values and left alone.
Symptom: a slide listing every assumption with a green, amber, or red dot and a recommendation to "monitor closely." Why it fails: attention is the scarce resource; a list of nine watch items produces zero watch items, and it lets the person who built the model avoid the risk of naming which numbers the conclusion actually rests on. Corrective: rank by swing, name the top two out loud, state explicitly that the rest are held fixed at measured values, and put a dashboard on the two you named and on nothing else.
The most common fate of a cost model is that it is built for a launch review, wins the argument it was built for, and is never opened again. Six months later someone finds it, and its false authority does more damage than its absence would have — because a spreadsheet with real arithmetic in it looks true in a way a rough guess does not.
A maintained model needs two things the launch-review version does not. The first is an assumption registry: every input listed with its value, its source, the date it was verified, and the trigger that forces a re-check. Prices are verified against the provider's pricing page and re-checked on any provider announcement. Token counts are measured from production and re-checked on any prompt or retrieval change. Behavioral rates — drafts per contact, cache hit rate — come from dashboards and are re-checked monthly, or immediately after any release that touches the prompt.
The second is a set of explicit re-run triggers, written down where the team can see them: a provider price change, any pull request that modifies the prompt or the retrieval configuration, a shift in the usage distribution beyond a stated threshold, and any pricing or contract decision that will cite the model. That last trigger is the one with teeth. If a number from this model is going into a customer contract, the model gets re-run that week, and the version that goes into the contract carries the date.
You already know how to do this; it is the discipline of a memo whose facts have to survive cross-examination. Every assertion is sourced, every source is dated, and the parts that rest on someone else's representation are flagged as such so a reader can weigh them separately. A cost model built to that standard is legible to a skeptical CFO in the same way a well-cited brief is legible to a skeptical judge: not because they trust you, but because they can check any single line without redoing the whole thing.
Every serving strategy you will ever be pitched is a position on the same three-way constraint. Route to a cheaper model and you spend quality to buy cost. Batch and you spend latency to buy cost. Cache and you buy cost with neither, which is why caching should always be the first thing you do and why it is nearly always the last thing anyone implements. Sample the model five times and vote, and you spend cost and latency to buy quality.
This module gives you the frame and then the two pieces of arithmetic that make the frame operational. The first is blended cost: how to compute what a cascade actually costs, and why that number is only as honest as the traffic shares in it. The second is cost per resolved outcome: the metric that reveals that both quality-maximalism and cost-minimalism are the same error viewed from opposite ends, and that supplies the economic argument for evals — you cannot find the minimum of a curve you have not measured.
For a fixed workload and a fixed set of available models, cost, quality, and latency form a constraint surface. You choose a position on it. You do not escape it, and the sentence "cheaper, better, and faster" describes either a genuine advance in the underlying models — which happens, but not on your roadmap's schedule — or a change in the workload itself, which is a different and often better conversation.
The value of naming the cost-quality-latency triangle is that it converts a stream of unrelated engineering proposals into a small set of moves with known trades. When someone proposes a change, ask which two corners it improves and which one it spends. If the answer is "none," they are either describing caching or describing a misunderstanding.
Model routing is the observation that your traffic is not homogeneous. Harbor's requests include straightforward password-reset replies, moderately involved billing explanations, and genuinely hard multi-policy disputes, and sending all three to the same frontier model means paying the price of the hardest one on every request. A cascade is the version of routing that does not require you to classify difficulty in advance: send everything to a cheap model first, evaluate the result against a confidence check, and escalate only what fails.
Harbor's cascade sends the draft request to the mid-tier model with the same cached prefix. At mid-tier rates the effective prefix rate is 0.70 × $0.10 + 0.22 × $1.00 + 0.08 × $1.25 = $0.39 per million, so the prefix costs $0.00234, the dynamic suffix $0.001, and the 400-token draft $0.002 — $0.00534, plus about $0.00025 for a fifty-token self-rating appended to the output. The cheap branch costs $0.00559. When the self-rating falls below threshold, the request is re-run on the frontier model at $0.01602, so an escalated request costs both: $0.02161.
Note what the escalation rate does to the pitch. Cascades are commonly sold with 60-to-80-percent savings figures, and those figures are achievable — but they assume both a large price gap between tiers and a low escalation rate. Harbor's saving is 43 percent rather than 75 percent for a specific and instructive reason: prompt caching had already captured most of the cheap win on the frontier path, so the cascade is competing against an optimized baseline rather than a list-price one. Optimizations do not stack the way their individual claims suggest, and the honest way to present a second optimization is against the post-first-optimization baseline.
A cascade's blended cost is only as honest as the traffic shares in it, and traffic shares drift. The escalation rate rises when the customer mix shifts toward harder contacts, when a knowledge-base update degrades retrieval, and — most insidiously — when someone tunes the confidence threshold upward to fix a quality complaint. That last one is a cost change disguised as a quality fix, and it will not appear in any quality review. Instrument the escalation rate, alert on it, and treat any threshold change as a pricing change.
Caching earns its place at the front of every optimization sequence because it is the only move that does not spend a corner. It reduces cost, leaves quality untouched by construction — the same tokens reach the model — and typically reduces time-to-first-token because the prefix does not need reprocessing. The cost is engineering discipline rather than product quality: the prefix must be genuinely stable, which constrains how you structure prompts and how casually you edit them.
Batching is the opposite: pure trade, no discipline required. You accept a completion window measured in hours and pay roughly half. The only design work is honest classification of your workloads by latency tolerance, and the classification is usually more permissive than teams assume — most products contain substantial offline work that was put on the interactive path by default rather than by decision.
| Harbor workload | Latency budget | Batchable | Reason |
|---|---|---|---|
| Interactive draft | Under 3s to first token | No | An agent is watching a spinner with a customer waiting |
| Escalation-risk classifier | Under 1s, blocks routing | No | Result gates which queue the contact enters |
| Nightly coaching notes | By 7am | Yes | Nine-hour window; nobody is waiting |
| Weekly KB gap analysis | By Monday | Yes | Multi-day window |
| Golden-set eval runs | Blocks a deploy | Partly | Pre-merge runs need speed; nightly regression sweeps do not |
Two of five Harbor workloads are fully batchable and a third is partly. At Harbor's scale the nightly coaching job alone processes every closed contact — 945 per seat-month across 500 seats — and moving it to batch pricing halves a five-figure monthly line. It required no product change and no quality argument. It required someone to ask which workloads own spare latency, which is a question nobody asks unless the triangle is in their head.
Here is the argument that ties the module together. Quality below a threshold does not save money. It moves money — out of the inference line and into human fallback, where the unit costs are one to three orders of magnitude higher. The right metric is therefore not cost per request but cost per resolved outcome: total spend, model plus human, divided by successful outcomes.
Harbor's numbers make this concrete. A draft that the agent accepts with light editing costs the model price and nothing else. A draft the agent rejects costs the model price plus roughly six minutes of the agent writing from scratch, which at $28 per hour of loaded time is $2.80 — over a hundred times the cost of the draft that failed. So cost per resolved contact is model cost per contact plus (1 − acceptance rate) × $2.80, and it is U-shaped in model quality.
Symptom: every model call in the product, including format validation, language detection, and routing classification, points at the frontier model, justified as "we don't want to risk quality." Why it fails: it pays the price of the hardest task on the easiest ones, which for a typical product is 50 to 70 percent of calls, and it does so without evidence — nobody measured whether the small model was worse at detecting a language. Corrective: route by task, and require an eval result rather than an intuition to justify each frontier assignment. "We didn't test it" is not a risk argument; it is the absence of one.
Symptom: a model downgrade shipped on the strength of the inference saving alone, with quality "spot-checked" rather than evaluated. Why it fails: it optimizes cost per request while the business pays cost per outcome. Downgrading Harbor from frontier to mid-tier saves $14.13 per seat-month and drops acceptance 17 points, which adds 945 × 0.17 × $2.80 = $450 per seat-month of agent rework. Corrective: no model change ships without a cost-per-resolved-outcome comparison, which requires the eval suite to report acceptance alongside spend.
This curve is the strongest business case for eval infrastructure that exists, and it is not the one usually made. Evals are typically justified as risk management: catching regressions before customers do. The stronger argument is that the acceptance rate is the x-axis of your cost curve, and a team without evals is not merely exposed to quality risk — it is structurally unable to find its own cost minimum, and will therefore sit at a randomly chosen point on a curve whose ends differ by 3x.
Everything so far has assumed a request with a beginning and an end. Agent loops break that assumption, and they break it multiplicatively: each iteration is a fresh billable request, and each iteration carries everything that came before it. The two multipliers compound, which is how a feature that costs a penny in the demo costs two dollars in production without anyone making a decision to spend two dollars.
This module works a second running example end to end: a ticket-triage agent that loses money on the average run and hemorrhages at the tail, costed honestly and then redesigned to a positive margin with the arithmetic attributed remedy by remedy. Along the way it makes an argument that belongs in your PRD template — that a step cap and a token ceiling are product requirements with the same standing as an authorization boundary, because an unbounded loop is an unbounded liability and the person who decides what happens when the budget runs out is deciding a user experience, not an error path.
Agentic amplification is the compounding of two independent growth terms. The first is step count: an agent that reads a ticket, searches a knowledge base, checks an order status, reads a refund policy, and writes a recommendation has made five billable requests where a single-shot feature made one. The second is context accumulation, which you met in Module 2 — but in an agent loop it is far more aggressive than in a conversation, because what accumulates is not a user's short message but the model's own reasoning plus the full verbatim output of every tool it called.
Multiply them and cost grows roughly quadratically in step count. Concretely: if each step adds G tokens to the context, then step k sends a base plus (k−1)G tokens, and the sum over n steps includes a term proportional to n². Doubling an agent's average step count does not double its cost; it roughly quadruples it. This is the single most important sentence in the module, because "let's let it try a few more times" is the most common product suggestion in the category.
Step count enters agent cost quadratically, not linearly. Any product decision that raises the average number of iterations — a more thorough search, an extra verification pass, a retry on ambiguity — costs far more than it looks like it costs, and the estimate you form by pricing one step and multiplying is not merely low, it is low by a factor that grows with how well the feature works.
Meet the second running example: a ticket-triage agent that reads an incoming support contact, gathers what it needs, and produces a triage summary — category, priority, suggested owner, and a draft first response — before a human sees the ticket. The business case priced its value at $0.60 per ticket, from the handle-time reduction it produces in the first-touch queue.
The naive design, which is what shipped: frontier model on every step, no step cap, full history in context, tool results appended verbatim. The measured shape is a 10,000-token base context (2,500-token system prompt, 3,500 tokens of tool schemas, 4,000 tokens of ticket thread), 7,000 tokens added per step (1,200 of agent reasoning and tool call, 5,000 of verbatim tool result — the knowledge-base search returns whole articles — and the 800-token prior assistant turn), 800 output tokens per step, and a mean of 12 steps.
At frontier rates, step k costs (10,000 + 7,000(k−1)) × $3/M + 800 × $15/M. Step 1 costs $0.042. Step 12 costs $0.273 — 6.5 times step 1 for an iteration that does the same kind of work. The mean run totals $1.89 against $0.60 of value: a loss of $1.29 per ticket, before any of the other COGS lines from Module 1.
The tail is worse than the mean, and the tail is where these features actually fail. The p95 run takes 22 steps and costs $5.78 — more than three times the mean and nearly ten times the value it produces. Because cost is quadratic in steps, the distribution of cost is far more skewed than the distribution of step counts, so any reasoning that starts "on average the agent takes about" is reasoning about the wrong statistic.
The mean run cost is the number that goes in the business case and the p95 is the number that shows up on the invoice, because expensive runs also correlate with the hard, high-value, most-visible tickets. A feature can be simultaneously profitable in aggregate and catastrophic on the accounts you most want to keep — and if a single enterprise tenant's ticket mix skews hard, their cost per ticket can sit at your p95 permanently while the fleet mean looks healthy.
An unbounded loop is an unbounded liability, and "unbounded liability" is not an engineering concern — it is a product decision that has been made by omission. A budget guard belongs in the PRD, specified with the same precision as an authorization rule, because the interesting part of a guard is not the limit but the behavior at the limit, and that behavior is a user experience.
A complete guard has five parts. A step budget: a hard cap on iterations. A token ceiling: a cap on total tokens consumed by one run, which catches the case where few steps each carry enormous context. A soft-degrade threshold, crossed before the hard limit, at which the run continues in a cheaper mode. The degrade behavior itself. And the hard-halt behavior, including what the user sees.
The budget guard is the economic sibling of the permission boundary, and the reasoning is identical. In both cases you are bounding what an autonomous process can do on your behalf, in both cases the bound must be enforced by the runtime rather than requested of the model, and in both cases the interesting design work is the behavior at the boundary rather than the boundary's location. An agent that asks permission before spending is as unreliable as an agent that asks permission before acting — the ceiling has to be a property of the loop, not an instruction in the prompt.
The same agent, rebuilt with four changes and nothing else. No new model, no new capability, no reduction in what it is asked to do.
A 6-step budget with the guard from Figure 5.2. Tool-result summarization: a small model compresses each tool result to 300 tokens before it enters context, so the per-step growth falls from 7,000 to 2,300 tokens. Caching of the 8,000-token stable prefix (system prompt, tool schemas, and the fixed ticket context). Routing: the mid-tier model handles the five tool-selection steps, where the work is choosing which tool to call next, and the frontier model is reserved for the single final synthesis step, where the judgment actually is.
The result is $0.1366 per mean run against $0.60 of value — a margin of $0.46 per ticket where there was a loss of $1.29. And the p95 is no longer a matter of hope: it is bounded by construction at the token ceiling, which is what a guard buys that an optimization cannot.
Symptom: a post-incident review of a runaway agent produces a fix for the specific loop that ran away — a better termination condition on that one tool, an extra check in that one branch. Why it fails: it treats an economic invariant as a defect. The next runaway will have a different shape, and the fix does not generalize because it was written against a shape rather than against a bound. Corrective: ship the specific fix and then institute the ceiling, so the next runaway — which is coming, and which nobody will predict — is bounded by construction rather than by having been anticipated.
Pricing an AI feature is where the cost model stops being an engineering artifact and becomes a commercial one. The four available structures are not a matter of taste, and the choice among them is not primarily about willingness to pay — it is about which usage distribution you can survive. Every structure works beautifully for some customers and fails predictably for others, and the discipline is knowing in advance which customers break yours and what you will do about them.
The frame that makes the four comparable is the relationship between two things: what the invoice counts, and what the customer's value scales with. When those move together, pricing feels fair to both sides and margin is stable across the user distribution. When they diverge, one side is systematically overcharged and the other systematically underserved — and in AI features the divergence is not symmetric, because the underpriced tail is also the expensive tail.
A price metric is what the invoice counts: seats, drafts, credits, resolved tickets, gigabytes. A value metric is what the customer's realized benefit actually scales with. They are different objects, and pricing is the act of choosing a price metric that tracks the value metric closely enough that both parties can live with the gap.
For Harbor the value metric is reasonably clear: value scales with resolved customer contacts, because the benefit is agent time saved and faster first responses. The candidate price metrics track that with varying fidelity. Seats track it loosely — a seat produces contacts, but a busy agent produces five times what a quiet one does. Drafts track it well, since drafts are roughly proportional to contacts. Resolved-without-escalation tracks it almost exactly, at the cost of being the hardest thing to attribute.
The diagnostic question for any pricing proposal is: when a customer gets twice the value, does the invoice roughly double? If yes, the structure is aligned and margin will be stable across your user distribution. If the invoice stays flat while value doubles, you have handed your best outcome to your worst margin. If the invoice doubles while value stays flat, the customer will notice, ration usage, and eventually leave — and rationing is the worse outcome, because it suppresses adoption of the feature you built.
In zero-marginal-cost software, a misaligned price metric costs you revenue you might have captured. In AI software it costs you money you have already spent. That asymmetry is why pricing an AI feature is a cost-model exercise before it is a market-comparison exercise: you cannot price against a competitor's number until you know which of your customers that number loses money on.
All four are worked against Harbor's pinned cost model — $0.01602 per draft — at three usage levels: a light user at 300 drafts a month, the median agent at 1,323, and a heavy user at 4,000.
Model cost figures below derive from the illustrative rate card pinned in Module 2 and the assumption table in Module 3. Before any of these margins are used to set a real price, re-verify the provider's rates on their published pricing page and re-run the model with the date recorded. A pricing decision made on a stale cost model is the most expensive kind of stale, because prices are sticky and costs are not.
Seat pricing at $40 per agent per month. Light user: $40 revenue against $4.81 of model COGS, an 88 percent margin. Median: $40 against $21.19, 47 percent. Heavy: $40 against $64.08, a negative 60 percent margin — the heavy user costs 1.6 times what they pay. The structure is maximally predictable for the buyer and maximally exposed for you, and the exposure is concentrated in exactly the customers who love the product most.
Usage pricing at $0.05 per draft. Light: $15.00 against $4.81, 68 percent. Median: $66.15 against $21.19, 68 percent. Heavy: $200.00 against $64.08, 68 percent. Margin is constant by construction, which is the entire point — but look at the heavy user's invoice. A support manager who budgeted $40 a seat now receives a $200 line for one agent, and the fact that it was earned does not make it survivable in a procurement conversation.
Credits at $45 per 1,000-draft pack, expiring monthly. Light user buys one pack, uses 300 drafts: $45 against $4.81, 89 percent — of which most is breakage — the 700 credits (70 percent of the pack) paid for and never redeemed. Median buys two packs, uses 1,323: $90 against $21.19, 76 percent. Heavy buys four packs: $180 against $64.08, 64 percent. Credits are usage pricing wearing a costume: they smooth the invoice into predictable purchases and add a margin subsidy from breakage, and in exchange they inherit every obligation of usage pricing plus a new one — the burn rate is now a number you publish and are held to, and quietly changing how many credits a draft costs is the fastest way to lose a customer's trust in the whole bill.
Outcome pricing at $0.75 per incremental first-contact resolution. Harbor lifts first-contact resolution from 58 to 66 percent, measured against a held-out control group of agents. Light user (214 contacts): 17 incremental resolutions, $12.86 against $4.81, 63 percent. Median (945 contacts): 76 incremental, $56.70 against $21.19, 63 percent. Heavy (2,857 contacts): 229 incremental, $171.43 against $64.08, 63 percent. Margin is stable and the price metric is the value metric — which is why this structure is both the most defensible and the least commonly shipped. Everything rests on the phrase "measured against a held-out control group," and maintaining a permanent holdout means deliberately withholding your product from some of the customer's agents forever, which you must be able to explain and they must agree to.
| Structure | Light (300) | Median (1,323) | Heavy (4,000) | Margin stability |
|---|---|---|---|---|
| Seat, $40 | $35.19 (88%) | $18.81 (47%) | −$24.08 (−60%) | Collapses at the tail |
| Usage, $0.05/draft | $10.19 (68%) | $44.96 (68%) | $135.92 (68%) | Constant |
| Credits, $45/1,000 | $40.19 (89%) | $68.81 (76%) | $115.92 (64%) | Declines with sophistication |
| Outcome, $0.75/lift | $8.05 (63%) | $35.51 (63%) | $107.35 (63%) | Constant, if attributable |
Every structure has a characteristic break, and the mature version of this decision is choosing the break you can operationally survive rather than pretending you found the one without a break.
Seat pricing breaks at the heavy tail. Revenue is capped and cost is not, so margin is a decreasing function of engagement, and the customers who adopt hardest are the ones who cost most. This is the failure mode that hurts most because it inverts every instinct you have: your power users become your worst accounts, your success metrics and your margin metrics point in opposite directions, and the natural response to a margin problem — drive adoption — makes it worse.
Usage pricing breaks on unpredictability. The meter transfers cost risk to the customer, and customers respond to unpredictable bills the way anyone does: they ration. A support manager who cannot forecast next month's Harbor invoice will tell agents to use it "only when needed," which suppresses adoption of the feature you built, degrades the outcome you were selling, and produces a churn conversation about value rather than about price.
Symptom: an unlimited-runs plan at a flat monthly price on a feature whose per-run cost is variable, unbounded, and user-triggered. The tell is a margin that is inversely proportional to engagement — your happiest cohort is your worst-performing one — and a finance team that cannot forecast COGS from bookings. Why it fails: flat rate is a bet that the usage distribution has a thin tail, and agentic usage distributions have the fattest tails in software: a small cohort scripting the agent against their whole ticket backlog can run it a hundred times a day for the price of one seat. Corrective: meter it, cap it, or price the tail. Concretely: publish an included allowance that covers the 90th percentile, meter overage at a rate that covers marginal cost with margin, and enforce a hard per-run budget guard (Module 5) so the worst individual run is a number you chose. All three, not one.
In practice the durable answers are hybrids, because a hybrid lets you take each structure's strength where it applies and cap each one's break where it bites. The common and defensible shape is a seat price with an included allowance and metered overage: predictable for the buyer across the range they actually operate in, aligned with value at the tail where alignment matters, and bounded on your side.
The decision procedure, in order. First, model the usage distribution — not the average, the shape, with a p90 and a p99. If you do not have production data, this is the single strongest argument for a metered beta. Second, price the tail explicitly: decide what happens at p99, whether that is overage, a hard cap, a degrade to a cheaper mode, or a conversation with the account. Not deciding is deciding that the tail is free. Third, pick the structure whose break you can operationally survive — meaning you have both a mitigation and someone who owns it. Fourth, plan the migration you will eventually run, because pricing changes are the hardest changes a product makes and the ones most likely to be forced by a cost structure you set before you had data.
Outcome pricing is a contract-drafting problem before it is a pricing problem, and you have drafted this exact clause under a different name. "Resolved without escalation" has to be defined with enough precision to survive a dispute in month eight, when the customer's own agents claim the resolutions and your attribution says otherwise. Who measures it, from what system of record, on what cadence, with what audit right, and what happens when the two sides disagree — that is a definitions section and a dispute-resolution mechanism, and shipping outcome pricing without one is shipping an agreement to argue later.
So far cost has been a function of what you send. This module is about what you can send, and when — the capacity side of the ledger, where a provider's rate limits and commitment offers are the mechanism by which they allocate a scarce resource and you decide how much of it to pre-purchase.
Two things make this a product conversation rather than an infrastructure one. First, a rate limit is a constraint on product behavior: what happens when you hit it is a user experience someone must design, and if nobody designs it, users discover it as an error. Second, a throughput commitment is a lease with a breakeven, and the arithmetic that decides whether to sign it is the same arithmetic you would apply to any capacity purchase — with one twist specific to this domain, which is that your own optimizations change the breakeven under your feet.
Providers meter throughput in tokens per minute and requests per minute, tiered by account standing and spend. The tier is a capacity allocation, and it binds your product in ways a price does not — you cannot pay a little more at 9:04am to serve the request in front of you. The limit is a wall, and a request that hits it returns a 429.
The product question is therefore not "how do we avoid rate limits" but "what does the product do at the limit," and there are four honest answers, each of which is a different experience. Queue: hold the request and serve it late, which is right when the work is not synchronous with a human. Degrade: serve from a smaller model or a lower-quality path that has separate capacity, which preserves the interaction at reduced quality. Shed: decline the request with a clear message and a retry affordance, which is honest and occasionally correct. Reserve: allocate capacity by tier or workload so the important paths never contend with the batch job — which is the one that requires deciding, in advance, which of your workloads is allowed to fail first.
Harbor's answer is a combination, decided at design time. The escalation-risk classifier and the interactive draft path draw from a reserved allocation; the nightly coaching job draws from what is left and queues indefinitely. If the draft path exhausts its allocation at a regional peak, Harbor degrades to the mid-tier model — the same cascade path from Module 4, reused as a capacity valve — and only sheds if that too is saturated, with a message that names the situation rather than showing a generic error. None of this is infrastructure work. All of it is product specification that infrastructure implements.
429s at peak are routinely filed as reliability incidents and routed to an on-call engineer, who raises the tier or adds retries and closes the ticket. Both responses are reasonable and neither is the decision that was actually required, which is a product one: this workload's demand exceeded the capacity we bought, so we either buy more, use less, or decide which requests lose. Filing it as an infra bug means the tradeoff gets made by whoever is awake, using the lever nearest to hand.
Provisioned throughput is a capacity commitment: you pre-purchase a sustained rate for a term and pay for it whether you use it or not. It is a lease, and like any lease it has a breakeven utilization — the fraction of the committed capacity you must actually consume for the commitment to beat paying on demand.
The arithmetic is simple and worth doing explicitly. Suppose a unit of provisioned throughput delivers 50,000 tokens per minute for $3,250 a month. Fully utilized, that unit passes 50,000 × 60 × 24 × 30 = 2,160 million tokens. At Harbor's effective blended on-demand rate of $2.165 per million — which already reflects prompt caching — those tokens would cost $4,676. So the commitment is worth signing above a utilization of $3,250 ÷ $4,676 = 69.5 percent, and below it you are subsidizing your provider.
Now apply it to Harbor at fleet scale. Five hundred seats generate 661,500 drafts a month, about 4.90 billion tokens, which averages 113,300 tokens per minute over the full month and would cost $10,597 entirely on demand. But the average is not the shape. Harbor's customers run follow-the-sun support across three regions, so the aggregate curve has a genuine 24-hour floor around 55,000 TPM and peaks near 260,000.
At some scale, self-hosting an open-weights model competes with API pricing. The memo that argues for it usually compares one number against one number, and the comparison is always favorable, because compute is the only cost self-hosting makes visible.
Harbor's version: two reserved GPU nodes sized to peak throughput at roughly $6,400 a month against $10,597 of API spend — "40 percent cheaper." The honest comparison adds the costs that the API price already includes and self-hosting moves onto your payroll. Ops engineering: serving infrastructure, autoscaling, upgrades, incident response — half a senior platform engineer, fully loaded, is about $10,400 a month, and half is optimistic in year one. Evaluation and safety infrastructure: the provider's model arrives with safety tuning and published evals; yours arrives with weights, so budget roughly $2,000 a month for the harness and the human grading that keeps it honest. Model refresh: the frontier moves, and staying within a generation means re-evaluating, re-tuning, and re-validating on a cadence — amortized, about $1,500 a month. Utilization risk: the hardware is sized to peak and paid for at every hour, whereas the API bills what you send; at Harbor's 38 percent average utilization against peak, more than half the committed capacity is idle.
Total: about $20,300 a month against $10,597 — roughly 92 percent more expensive, and the advantage inverted the moment the memo included the costs the API price was quietly paying for.
The interesting question is not which is cheaper at Harbor's scale but where the crossover sits, because the fixed costs are fixed and the compute scales. With about $13,900 a month of volume-independent cost and compute at roughly 60 percent of the API's rate, self-hosting overtakes at about 3.3 times Harbor's volume — call it 1,700 seats. That is the number the memo should lead with, and the honest version adds two qualifications: the crossover moves up as your caching and routing lower the effective API rate, and it does not price the option value you give up, which is the ability to switch models next quarter when a better one ships.
Everything past the decision memo — GPU cluster architecture, inference-server tuning, batching and paged attention, the economics of training or fine-tuning a model of your own — is out of scope for this course. What is in scope is the shape of the comparison: a total-cost-of-ownership analysis with a stated crossover volume, not a GPU-hour price set against an API price. If a build-versus-buy memo crosses your desk without ops, evals, refresh, and utilization risk in it, it is not a memo about cost.
Everything in this course has been a decision made in advance. This module is about the machinery that tells you whether the decisions held — because they will not, and the interval between a cost regression and your discovery of it is the interval during which you pay for it.
Three pieces of machinery, in order of how often they are missing. Unit costs as service-level indicators: tracked as distributions, on the same dashboard as latency and error rate, with thresholds someone is paged about. Attribution: request-time tagging that lets you answer which feature, which workflow, and which tenant, because the invoice answers none of those. And a cost gate in CI, so that a prompt change which doubles token spend fails a build instead of surfacing four weeks later in a number nobody can decompose. The module closes with the field catalog of every pathology the course has named.
Your dashboard has latency and error rate on it because those are the numbers that tell you whether the service is doing its job. Cost per action belongs beside them, for the same reason and with the same treatment: as a distribution, not a total, with p50 and p95, with a threshold, and with someone whose job it is to look.
A unit-cost SLI differs from a spend total in three ways that matter. It is per unit, so it moves when the design changes rather than when volume changes — a 30 percent traffic increase leaves it flat, which is exactly what you want, because volume growth is not the event you are watching for. It is a distribution, because the p95 is where budget guards bind and where per-tenant losses hide. And it is fast, computed from request-time metering within a minute or two rather than from a bill that closes monthly.
Harbor tracks three. Cost per draft, p50 and p95, alerting when p50 exceeds $0.020 for thirty minutes — a 25 percent headroom over the $0.016 baseline, which absorbs normal cache variance and catches a prompt regression the same afternoon. Cost per seat-day, which composes the per-draft cost with the volume behavior and catches the regeneration-rate rise that a per-draft metric would miss entirely. Cost per tenant-month, projected daily rather than measured at close, which is the number that finds an account whose ticket mix is quietly running at your p95.
Symptom: the first person to notice a cost change is in finance, and the artifact they noticed it in is an invoice. The tell is the sentence "our AI spend was up 40 percent last month, does anyone know why?" — a question that is unanswerable from the artifact that prompted it. Why it fails: a monthly bill has a thirty-day detection window and no dimensionality. By the time it arrives you have paid for a full month of the regression, and the bill cannot tell you which change caused it because it aggregates every cause into one number. Corrective: unit-cost SLIs computed from request-time metering, with alert thresholds and a named owner. The bill becomes a reconciliation artifact — you check that it matches what your metering already told you — rather than a discovery artifact.
Cost attribution is the ability to answer "who spent this, on what, doing which thing." It is a request-time discipline, and it cannot be retrofitted, because the information it requires exists only at the moment of the call and is not present in what the provider bills you.
The mechanism is unglamorous: every model call carries tags, and the metering layer records tokens and computed cost against them. Harbor tags five dimensions. Tenant — the customer account, which is the one finance and account management ask for. Feature — draft, classifier, coaching-note job, eval run — so a spend change can be localized to a surface. Workflow — the user-visible operation the call belongs to, which is what lets you say "a contact costs this much" rather than "we made this many calls." Model and route — which tier served it, and whether it came through the cheap branch or an escalation, which is how you monitor a cascade's traffic shares. Cache outcome — read, miss, or write, which is how the hit rate becomes a measured variable rather than an assumption.
Everything you know about the golden signals applies here unchanged; cost is simply a fourth one with a dollar sign. It is a rate, it has a distribution, it needs cardinality-controlled labels, and it degrades under exactly the conditions the other three do. The one discipline this signal adds is that its labels have commercial consumers — finance, account management, and the pricing decision from Module 6 — so the tag schema is a cross-functional contract rather than an engineering convenience, and changing it silently breaks reports outside your team.
Prompt changes are token changes. A retrieval tweak, a clarifying instruction, an extra few-shot example, a richer tool schema — each of these edits the size of what you send on every request forever, and none of them looks like a cost change in a pull request. So put the cost in the pull request.
The mechanism composes with infrastructure you should already have. Your eval suite runs a golden set on every change and reports a pass rate. Have it also report cost per run, computed from actual token counts at current rates, and fail the build when cost per run exceeds its budget — Harbor uses ±15 percent against the baseline recorded on the last merge to main. A change that trips the gate is not blocked forever; it is blocked from merging silently. The author either brings the cost back down or updates the baseline with an explicit justification that a reviewer approves, which converts an invisible drift into a decision with a name attached to it.
This is the same gate discipline as quality evals, pointed at spend, and it should live in the same job for a practical reason: it costs nothing extra. You are already running the golden set and already paying for those tokens, so counting them and comparing to a budget is a few lines of reporting. A team with eval infrastructure and no cost gate has built ninety percent of the machinery and declined the last ten percent.
Here is every anti-pattern this course has named, consolidated for field recognition. Each is stated as a symptom you can observe from outside, because that is how you will actually encounter them — in someone else's plan, in a review, in a number that does not sit right.
| Pathology | Symptom in the wild | Corrective | Taught in |
|---|---|---|---|
| Turn-one cost modeling | A per-request cost measured once, multiplied by an average session length | Sum the growing series; sanity-check against observed p50 and p95 session lengths | m2 |
| Cache hit rate as a constant | A hit-rate figure typed into a model cell with no dashboard beside it | Carry it as a monitored range with its low end from the worst tenant; alert on every deploy | m2 |
| Sensitivity as inventory | Nine variables, all flagged "worth monitoring" | Rank by swing, filter by what you can influence, name two, dashboard only those | m3 |
| Quality-maximalism | Every call points at the frontier model, including format validation and routing | Route by task; require an eval result, not an intuition, to justify each frontier assignment | m4 |
| Cost-minimalism | A model downgrade justified by inference savings and a quality "spot-check" | Compare cost per resolved outcome, including human fallback, before any model change ships | m4 |
| Runaway loops as a bug class | Post-incident fix targets the specific loop that ran away | Ship the fix, then institute a step cap and token ceiling so the next one is bounded by construction | m5 |
| Flat-rate on unmetered agentic usage | Margin inversely proportional to engagement; COGS unforecastable from bookings | Published allowance at p90, metered overage, and a per-run budget guard — all three | m6 |
| Unit costs discovered in the bill | "Our AI spend was up 40 percent last month, does anyone know why?" | Request-time metering, unit-cost SLIs with thresholds, an owner; the bill becomes reconciliation | m8 |
Read down the corrective column and the same instruction appears in eight forms: decide the number in advance. Sum the series before you commit to the estimate. Set the cache-hit range before you build the model. Name the two variables before you build the dashboard. Establish the acceptance rate before you change the model. Set the ceiling before the runaway. Price the tail before you sign. Put the threshold on the dashboard before the invoice arrives.
Every pathology in the catalog is the same mistake in different clothing: treating cost as an output of the system rather than an input to its design. Cost as an output is something you observe, explain, and apologize for. Cost as an input is a number you chose, defended in a design review, and can point to in the architecture. The whole course is the argument that the second posture is available to you — that context length, model tier, cache design, loop budget, price metric, capacity commitment, and alert threshold are all decisions, and that the PM who makes them deliberately is running the meter rather than reading it.
This is the complete text of the course. With JavaScript enabled, this same page runs the interactive edition — a self-diagnostic that reorders the syllabus around your gaps, knowledge checks, applied worksheets with model answers, and a spaced-repetition review queue — with progress saved locally in your browser.