Under the Hood

A model is a function with dials on it. You hand it an input, it returns an output, and its behavior is governed by a set of numbers — the parameters, or weights — that a programmer never wrote by hand. In ordinary software, a developer states the rule: if score > 92 and price < 60: recommend. In a learned system, the shape of the rule is fixed in advance and the numbers inside it are chosen by an optimization procedure that has seen examples.

Take the toy problem this course will keep returning to: predicting how a particular member of a wine club will rate a bottle. Start with the simplest possible parameterization — a weighted sum of a few features:

predicted_rating = w1*critic_score + w2*price + w3*years_in_bottle + b

There are four numbers here (three weights and an intercept). Training means searching for the four numbers that make the predictions closest to the ratings members actually gave. Nothing in that sentence requires intelligence, intention, or understanding; it requires a definition of closest and a procedure for searching. A frontier language model differs from this toy in that it has hundreds of billions of parameters arranged in a far richer functional form, and its inputs are token sequences rather than three columns. It does not differ in kind.

This is why the productive question in a vendor meeting is never "how does the model work." It is "what were the labels." A model is a compressed summary of the examples it was fit to, filtered through the objective it was scored against. If you know both, you can predict most of its failures before you see one.

You cannot search for good parameters until good is a number. The loss function is that number: it takes the model's predictions and the true answers and returns a single scalar that training will try to make small. For the rating predictor, the default choice is squared error — average the squared gap between predicted and actual rating across all training examples:

loss = mean( (predicted_rating - actual_rating)^2 )

Squaring is not a formality. It declares that one prediction off by 4 points is worse than four predictions off by 1 point each — 16 versus 4. That is a value judgment about which mistakes hurt, and it was made by whoever chose the loss, which in most organizations is a machine learning engineer under deadline, not the person accountable for the product.

Read this way, choosing a loss is a policy act with the same structure as choosing a penalty schedule. In a fraud screen, whether you weight a missed fraud at 10 times a false alarm or at 500 times is not a modeling detail — it fixes the operating posture of the entire product, and it is legible in one line of the training code. Ask for that line.

Given a loss, how do you find the parameters that minimize it? For the four-parameter wine model you might imagine trying combinations at random, but that strategy dies immediately at scale: a model with a million parameters has a search space no enumeration can touch. Gradient descent replaces search with a local rule that is almost embarrassingly simple.

Freeze every parameter but one and ask: if I increase this weight a hair, does the loss go up or down, and how fast? That slope is the gradient with respect to that weight. Now step the weight a small distance in the downhill direction. Do this for every parameter at once, on a batch of examples, then repeat with the next batch. That is the entire training loop. The learning rate is the size of the step.

Two failure modes fall straight out of the picture. A learning rate that is too large overshoots the valley and the loss diverges, often within a handful of updates; you see it as a loss curve that climbs to infinity or produces NaN. A learning rate that is too small converges honestly but consumes a training budget you do not have, and can stall in a shallow dip well above the good solutions. In practice, teams schedule the rate — larger early, decayed later — which is exactly the intuition the diagram gives you.

Here is the fact that makes the whole rest of the course necessary. A model with enough capacity can drive its training loss to zero on any dataset — including a dataset whose labels were assigned by coin flip. It does this not by discovering structure but by storing the answers. With more parameters than training examples, that storage is trivially available.

So there are two ways to score perfectly on data you have seen. One is generalization: the model found regularities that compress the data — old-world reds with high acidity and modest alcohol get rated well by this member — and those regularities extend to bottles it has never encountered. The other is memorization: the model stored a lookup table and extends to nothing. On the training data, these look identical. Byte for byte, metric for metric, identical.

The compression framing is the useful one. A model that genuinely learned has represented ten thousand ratings in far fewer effective bits than the ratings themselves, and the compression is only possible because there was structure to exploit. A model that memorized has spent bits proportional to the data. Compression is evidence of understood structure — which is also why a model that is too small for its task underfits: it lacks the bits to hold even the real regularities.

Overfitting is not a synonym for a bad model. It names a specific event with a specific time signature: the moment training continues to improve performance on data the model has seen while performance on data it has not seen starts getting worse. Before that moment, more training helps. After it, more training actively hurts — the model is spending its remaining capacity on the noise in the training set, and noise does not repeat.

The structural problem is that the training curve alone cannot show you this. From inside training, everything looks like progress: the loss goes down every epoch, monotonically, right through the point where the model started harming itself. The only way to see the divergence is to hold data back and measure on it as you go.

Three consequences follow. First, early stopping — keeping the checkpoint at the validation minimum rather than the final one — is not a trick; it is the direct operational reading of this curve. Second, a team that reports only training loss has not withheld the interesting number, they have failed to compute it. Third, and most usefully for you: if someone shows you a single loss curve going down and calls it evidence of learning, the correct question is simply "what is the other curve doing."

Data leakage is any path by which information from the evaluation data reaches the model — directly, through a feature, or through the humans tuning it. Leakage does not produce error messages. It produces excellent numbers, which is what makes it the most dangerous failure mode in applied ML: the symptom of leakage is success.

There are four paths, and they are worth memorizing as a set because each has a different remedy.

The three-way split is standard: the training set fits the parameters, the validation set guides every human and automated decision (architecture, features, hyperparameters, early stopping, thresholds), and the holdout or test set is measured once, at the end, to produce an honest estimate of performance on unseen data. Typical proportions are 70/15/15, but proportions matter far less than the discipline governing the third partition.

The useful frame is evidentiary. The test set is admissible only for as long as its chain of custody is intact. Every time a decision is made in light of test results, the sample has been contaminated by the process it was meant to evaluate — and unlike physical evidence, the contamination leaves no mark. The number still prints. This is why serious teams write the policy down: who may run the test evaluation, how many times, and what happens when the result is disappointing (the answer is not "iterate against it").

A product demo is a curated performance on selected inputs, prepared by people who know which inputs fail. That is not an accusation; it is what a demo is. The queries were chosen because they work, the documents were chosen because the model handles them, and any example that embarrassed the system during rehearsal was removed. In the vocabulary of this module: a demo reports training-set performance, whatever data it happens to run on.

This means the appropriate response to an impressive demo is neither belief nor cynicism. Both are unpriced. The appropriate response is a distribution request, and it has a standard form: run this on a sample I select, from my data, with my labels, and let me see the failures. A vendor who can do this within a week has a real product. A vendor who cannot has told you something more informative than the demo did.

The same discipline applies internally, and this is where product managers most often lose the thread. Your own team's demo to the executive review is subject to identical selection pressure — the engineer chose the five examples that morning, and chose them by running twenty. Ask for the twenty.

Computers cannot compare meanings; they compare numbers. An embedding model is a neural network trained so that the vector it produces for an input sits near the vectors of related inputs. The output is typically a few hundred to a few thousand floating-point numbers — 768, 1,536, and 3,072 dimensions are common — and no individual dimension has a human-readable meaning. The information lives in the relative positions.

The crucial question is: near in what sense? The answer is never "in the sense of meaning." It is always "in whatever sense the training objective rewarded." A model trained by pulling together text fragments that appeared close together in documents learns topical adjacency. A model trained on question-and-answer pairs learns that a question should sit near its answer — which are topically related but textually dissimilar. A model trained on product co-purchase data learns that a corkscrew is near a wine rack, which no text model would say.

This is why the choice of embedding model is a product decision with a testable criterion. For the wine cellar, a general-purpose text model trained on web documents will place a Chablis tasting note near a Sancerre note because they share vocabulary — mineral, citrus, saline. That happens to be useful. It will also place a note describing a wine's flaws near a note praising the same characteristics, because both mention brett and barnyard, and it has no encoding of valence. That is not useful, and you will only discover it by testing the model against pairs you have labeled yourself.

Given two vectors, three measures dominate practice, and they answer different questions.

The relationship worth internalizing: on vectors that have been normalized to unit length, cosine similarity and Euclidean distance rank neighbors identically, and dot product becomes cosine. Normalization is therefore the decision that collapses the choice, and most embedding providers normalize for you — which is why cosine is the default and why the metric debate is usually less consequential than teams expect.

What no metric can do is repair a mismatch between the model's notion of similar and your product's. If your users mean "same wine style at a lower price" and the model means "similar tasting vocabulary," switching from cosine to dot product changes nothing that matters. The fix is a different model, a fine-tuned one, or a hybrid with structured filters.

Here is the collapse. Every one of these features is nearest-neighbor lookup in an embedding space, differing only in what plays the role of query and what you do with the neighbors.

• Semantic search — query: the user's text, embedded. Neighbors: documents. Output: the neighbors, ranked.
• Recommendation — query: an item the user liked, or the average of several. Neighbors: catalog items. Output: neighbors minus what they have already seen.
• Clustering — query: none; instead find groups of points that are mutually near. Output: group assignments.
• Classification — query: the new item. Neighbors: labeled examples. Output: the majority label among the k nearest — a working classifier with no training step at all.

That last one deserves emphasis because it is the most under-used tool in an early-stage product. If you have 200 labeled support tickets and need routing today, embedding them and assigning each new ticket the majority label of its 15 nearest neighbors will often get you to 70–80% accuracy in an afternoon, with the enormous operational advantage that adding a new category means adding examples rather than retraining anything.

Make it concrete. Take 1,800 wines, each with a tasting note of 40–80 words. Embed every note; store the 1,800 vectors. Now a member says: something like the Chablis I loved, but I want to try something new. Embed that Chablis's note, find its nearest neighbors, drop what the member has already bought, return the top six.

What this does spectacularly well is the long-tail semantic match that keyword search cannot reach. The member's Chablis note says "oyster-shell minerality, taut, unoaked." A Muscadet from a producer whose note reads "saline, chiseled, no oak influence" shares almost no vocabulary with it and sits three positions away in the embedding space. No keyword system finds that. That single capability is the reason embeddings are in the product.

And there is the faceplant. The member's request was "under $50, 2015 or later, and not something I've already bought." None of those constraints exist in the embedding space. Price is not a direction you can travel; vintage is not a region. A tasting note that says "an astonishing value" will pull a $310 bottle toward the cheap neighborhood, because the space encodes the language, not the fact.

The correct architecture is the hybrid: apply structured filters first (a SQL WHERE clause, or the metadata filter your vector database provides), then rank the survivors by similarity. Filter for correctness, rank for relevance. Getting this order right is most of what separates a vector-search feature that ships from one that gets pulled.

Comparing a query against 1,800 vectors is trivial arithmetic. Comparing it against 40 million is not, and this is where embeddings become infrastructure. An ANN index — approximate nearest neighbor, in structures like HNSW or IVF — makes the query fast by giving up the guarantee that it returns the true nearest neighbors. Typical settings recall 95–99% of true neighbors at a fraction of the cost, and the tunable knob trades recall against latency and memory.

That approximation is usually fine and occasionally is not, and the distinction is worth stating in a spec. For discovery surfaces, missing one of the ten best matches is invisible. For a compliance screen where the whole point is that nothing similar to a known-bad document escapes, a 97% recall index has a defined miss rate, and you should either raise the recall setting or run an exact scan over the filtered candidate set. Approximation is a product decision.

Two more obligations round out the operational picture. First, corpora grow and shift, so the neighborhoods that made sense at launch drift; a scheduled retrieval-quality check on a fixed labeled query set is the monitoring that catches it. Second, storage is real: 40 million documents at 1,536 dimensions in 4-byte floats is about 245 GB of raw vectors before index overhead, which is a hosting decision, not a rounding error — and it is a large part of why dimensionality-reduced embedding variants exist.

A linear regression predicts by multiplying each input by a learned weight and adding the results. Its virtue is not accuracy; it is that the weights are readable claims. Fit willingness-to-pay against wine features and you get statements a human can accept or dispute:

willingness_to_pay = 11.4*(critic_score - 88) + 6.2*(years_in_bottle) - 0.9*(alcohol_pct) + 34.10

That first coefficient says each point of critic score above 88 adds about $11.40 to what this member segment will pay. A buyer can check it against what they know about the market and either nod or object — and either reaction is more informative than an accuracy metric. Logistic regression is the same instrument for classification, returning well-calibrated probabilities rather than a score, which matters whenever a downstream decision needs a threshold with an expected-cost calculation behind it.

The reason to insist on a linear model even when you intend to ship something else is the baseline. A number without a baseline is unpriced: 0.87 AUC is excellent for some problems and embarrassing for others, and the only way to know which is to see what a simple model gets. Three baselines are worth requesting by name — predict the majority class or the mean, predict the most popular items, and a logistic or linear regression on the obvious features.

A decision tree splits the data on one feature at a time — critic_score < 90?, then price < 45? — carving feature space into rectangular regions and predicting a constant within each. A single tree is weak and unstable. The insight that makes it powerful is boosting: fit a small tree, look at what it got wrong, fit the next small tree to those residual errors, add it to the ensemble, and repeat several hundred times. Each tree is a shallow correction; the sum is a precise, highly nonlinear function. That is a gradient-boosted tree ensemble — XGBoost, LightGBM, CatBoost — and it is the same gradient-descent idea from module 1, taking steps in function space instead of weight space.

Why does this beat deep learning on tables? Four reasons, all stateable in a meeting.

• Feature heterogeneity. Business tables mix dollars, counts, dates, and categories with hundreds of levels. Trees split on thresholds and are indifferent to scale; neural networks need every feature normalized and encoded, and remain sensitive to how you did it.
• Sample size. Business problems live at thousands to low millions of rows. Neural networks show their advantage in the regime where data is effectively unlimited relative to the pattern's complexity.
• No useful pretraining. Deep learning's dominance in text and images comes from transferring representations learned on enormous general corpora. There is no pretrained model for your subscription table's idiosyncratic columns.
• Irregular structure. Tabular relationships are full of thresholds and interactions — a discount matters below a price point and not above it — which is precisely what tree splits represent naturally and smooth networks approximate awkwardly.

The routing procedure is short enough to hold in your head, and its most important property is not the branches but the asymmetry it imposes: the burden of proof sits on the more expensive model, always.

Two clarifications keep the flowchart honest. First, most real problems are mixed — the wine recommender has structured purchase history and free-text tasting notes — and the right answer is usually to convert the unstructured part into features (embeddings compressed to a handful of dimensions, or model-derived scores) and feed them to the tabular model. Second, the flowchart routes the ranking or prediction task. It says nothing about the presentation layer, which is where LLMs earn their keep.

Now the comparison the module promised. The task: given a member, rank the current 1,800-bottle allocation and pick six. Three approaches, one problem, assumptions on the table first — because a comparison whose assumptions are hidden is a marketing exercise.

Read the numbers before drawing the conclusion. The GBM wins ranking quality per dollar decisively where labeled history exists — it is 300 times cheaper than the LLM and ranks better. The LLM ranks worse than the cheap model while costing dramatically more, which is the result most teams do not expect and the reason this comparison is worth running rather than assuming. Embeddings are mid-quality and mid-cost, but they are the only approach that does not depend on the member having a history, which is why they hold up in cold start where the GBM drops below the popularity baseline.

That architecture is worth stating explicitly because it is what the evidence supports and it is not what either enthusiasm or skepticism would have produced. The ranker is a boosted ensemble over structured features plus a handful of embedding-derived columns (similarity to the member's three highest-rated bottles, distance to their purchase centroid). Cold-start members are routed to pure similarity until they have five ratings. The LLM takes the ranked six and writes the note that says why this Muscadet, given that Chablis — the part of the product members actually quote back, and a job the cheap models cannot do at all. Each component does the thing it is best at, and the expensive component is used where its cost buys something the cheap ones cannot deliver.

Start from where a base model comes from. Pretraining ran gradient descent over an enormous general corpus, producing weights that encode broad linguistic and world structure. Fine-tuning resumes that same process, at a much smaller learning rate, on a curated dataset of your examples. The weights shift a little. The architecture does not change; no new storage is added; nothing is indexed.

Consider what it takes to make a model reliably reproduce a fact — say, that a particular regulation's compliance deadline moved to March 2027. To install that through weight updates you would need enough examples, repeated enough times, at a large enough learning rate, that the update survives all the competing pressure from every other example. Do too little and the model produces the old answer or an interpolation between the two. Do enough to be reliable and you have moved the weights hard enough to degrade neighboring capabilities. Meanwhile the fact will change again next quarter, and the only remedy is another training run.

Now consider making the model always answer in your firm's memo structure — issue, rule, application, conclusion, with citations in a house format. Two hundred well-constructed examples will do it, and the change is stable because it is a consistent pattern rather than a point fact competing against a mountain of prior evidence.

A full fine-tune updates every parameter. For a 7-billion-parameter model that means holding the weights, their gradients, and the optimizer's per-parameter state in memory simultaneously — in practice several times the model's own size, which is why full tuning of even mid-sized models means multi-GPU infrastructure. You also get a complete new model artifact per tuned variant: three customer-specific tunes means three full model copies to store, serve, and version.

LoRA — low-rank adaptation — takes a different route. Freeze the base model entirely. Alongside selected weight matrices, train small pairs of matrices whose product has the same shape as the original, and add that product to the frozen weight at inference. Because those matrices are deliberately low-rank (an inner dimension of 8 to 64 is typical), the trainable parameter count lands around 0.1–1% of the model's. The base model never moves; the adapter is a file of tens to hundreds of megabytes.

The operational consequences are larger than the cost savings. Adapters are swappable — one base model in memory, many adapters loaded per request, which turns per-customer tuning from an infrastructure program into a file management problem. And because the base weights are untouched, the blast radius of a bad tune is bounded: you unload the adapter.

The third family answers a different question. Supervised fine-tuning needs demonstrations: here is an input, here is the correct output, imitate it. But for many judgments — which of two summaries is more useful, which refusal is better calibrated, which explanation a member would find helpful — nobody can write the ideal output on demand, while anyone competent can rank two candidates in seconds.

Preference optimization trains on exactly that signal. Collect pairs where a human (or a model standing in for one) marked one response better, then adjust the model so preferred responses become more probable and rejected ones less so. RLHF does this through an intermediate reward model; DPO does it directly from the pairs, which is simpler and now more common in practice.

The critical property: preference optimization selects among behaviors the model can already produce. It shifts probability mass toward outputs already in the model's repertoire. If the model cannot write valid Terraform, no volume of preference data teaching it to prefer valid Terraform will make it able to — you would be rewarding a capability that is not there. Preference tuning is a taste transplant, not a skill transplant.

Before any of the three, run the triage. It has one question at the top, and getting it right saves more money than any other decision in this module: is the gap knowledge or behavior?

A knowledge gap means the model does not have the facts — your policies, your catalog, this quarter's regulations, the customer's account state. Retrieval fixes this properly: the facts live in a store you can update with a write, audit with a query, cite in the output, and permission per user. Tuning fixes it improperly, if at all.

A behavior gap means the model has whatever facts it needs but does the wrong thing with them — wrong format, wrong register, wrong task framing, wrong sense of when to hedge. Prompting fixes a surprising amount of this. Tuning fixes what prompting cannot, and buys back the context-window cost of a very long instruction block.

Three practical matters decide whether a tuning project succeeds, and none of them is the choice of algorithm.

Data quality dominates volume. For behavior shaping, several hundred carefully constructed examples routinely outperform tens of thousands of scraped ones. The reason follows from module 1: the model fits whatever regularity is in the data, so inconsistency in your examples is a regularity it learns — teach it that the format varies and it will faithfully vary the format. The expensive part of a tuning project is producing 300 examples that genuinely agree with each other, and that cost is measured in expert hours, not GPU hours.

Catastrophic forgetting is the silent regression. Aggressive tuning on a narrow objective degrades capabilities outside it. A model tuned hard on clause drafting gets worse at summarizing, at following unrelated instructions, at declining out-of-scope requests. Nothing announces this — the tuned task looks better, which is what everyone is looking at. It is caught only by evaluating capabilities you did not tune for, which nobody does unless the eval plan requires it.

The before/after evaluation is not optional. Every tuning run needs: a held-out set for the target behavior, a regression set covering general capability, and a fixed decision rule stated in advance. "It looks better" is the standard failure, and it fails in the specific way module 2 predicts — the examples someone looks at are the ones they remember struggling with, which is a training set.

A model's weights are numbers, and numbers need bits. Training typically uses 16-bit floating point, so each parameter costs 2 bytes. Quantization stores them in fewer bits — 8-bit integers at 1 byte, 4-bit at half a byte — by mapping the range of real values onto a coarser grid of representable ones. The memory saving is not an optimization to be hoped for; it is arithmetic.

memory for weights = parameter_count x bytes_per_parameter

7B model:   FP16 -> 14.0 GB    INT8 -> 7.0 GB    INT4 -> 3.5 GB
70B model:  FP16 -> 140 GB     INT8 -> 70 GB     INT4 -> 35 GB

Add roughly 15–25% on top for the KV cache, activations, and runtime overhead, scaling with context length and batch size. So on a single 24 GB card: a 7B model runs comfortably at FP16 (about 17 GB all-in), a 13B model needs INT8, a 34B model needs INT4 and is tight, and a 70B model does not fit at any precision that preserves useful quality. That last sentence is the one that ends most self-hosting proposals, and it takes fifteen seconds to derive.

What is lost is precision at the margins. Rounding each weight introduces a small error, errors accumulate through the layers, and the model's outputs shift slightly. Modern quantization methods reduce this considerably by calibrating on sample data and keeping the most sensitive layers at higher precision — but they reduce the loss rather than eliminating it.

The size of the quality cost is task-dependent, and this is the part people get wrong in both directions. On tasks with substantial redundancy — classification, tagging, summarization, extraction — INT8 is frequently indistinguishable from FP16 and INT4 costs a point or two. On tasks where a long chain of steps must all be right — multi-step reasoning, precise code generation, exact arithmetic — small per-step degradations compound and the drop can be severe. There is no universal quantization penalty. There is only your task's number, and you get it by running your evaluation set twice.

Distillation attacks size differently. Instead of compressing an existing model's weights, train a smaller model to reproduce a larger model's outputs — historically from scratch on the teacher's outputs, though for LLMs today the student is usually an already-pretrained base fine-tuned on them. The teacher generates outputs — often with its full probability distributions, which carry more information than the final answer alone — and the student learns to match them on that data.

What makes this work better than training the small model on the original labels is that the teacher's outputs are a richer signal. A ground-truth label says "this ticket is a billing issue." The teacher says "73% billing, 19% account access, 5% refund, 3% other," which encodes the similarity structure between categories, and the student learns from that structure.

The critical limitation is the boundary, and how sharp it is depends on how the student was built. The student learned to imitate the teacher on the distribution it was shown, and inside that region it can be remarkably close to the teacher at a fraction of the size. A student trained only to imitate on that slice has no fallback outside it — it never had the teacher's general capability, only an imitation of its behavior on a specific slice — so the degradation at the boundary is a cliff: not gradual, and it does not announce itself. When distillation instead fine-tunes an already-pretrained base (the common case for LLMs today), the student keeps that base's broad capability and degrades more gently past the taught slice. The narrow-slice case is the one to design around, because its failure is the one you cannot see coming.

The product consequence: a distilled model is a specialist you must fence. It belongs behind a router that sends anything unusual to the larger model, and behind a monitor that watches the input distribution for drift away from what it was taught — the same monitoring that module 8 requires for every deployed model, and more urgent here because a narrow-slice student's cliff is steeper.

Make it concrete on hardware you can touch. A single-GPU workstation with 24 GB of VRAM running a ROCm stack is a realistic home for a production-adjacent workload, and the arithmetic above tells you exactly what it will hold: a 7B model at FP16 with room for a long context, a 13B at INT8, a 34B at INT4 if you keep the context modest. Not a 70B, at any precision worth running.

Whether that constraint matters depends entirely on the task, which brings the module's final anti-pattern into view.

The self-hosting decision itself is then a straightforward comparison, provided you count honestly. On the hosted side: per-token cost times projected volume. On the self-hosted side: hardware amortization, electricity, the engineer-hours to build and maintain the serving stack, and the quality difference measured on your evaluation set. Self-hosting wins on sustained high-volume narrow tasks, on data that cannot leave your premises, and on latency floors that a network round trip cannot meet. It loses on bursty traffic, on broad task ranges, and — most often — on the maintenance line that nobody put in the model.

Each question falls out of something you already know.

The order matters in practice. Provenance and baseline can be asked in any meeting without preparation and eliminate a surprising share of weak claims immediately. Metric and leakage require a specific answer and often produce visible discomfort, which is itself information. Distribution is the one that survives every honest answer to the first four — a model can be genuinely good and still be useless on your data — and it is the question that converts an evaluation into a pilot design.

Now the claim you will actually be handed: our model detects compliance violations with 98% accuracy.

The claim is true. Work it.

Assume the population it was measured on: 10,000 documents, with a violation prevalence of 1% — so 100 actual violations and 9,900 clean documents. The model has 90% sensitivity (recall) and a 2% false-positive rate. Those are respectable numbers for this kind of system.

Actual violations:        100
  caught (true positives):   90     (90% sensitivity)
  missed (false negatives):  10

Clean documents:        9,900
  false alarms (FP):        198     (2% of 9,900)
  correctly cleared (TN):  9,702

accuracy  = (90 + 9,702) / 10,000 = 97.92%   -> "98% accuracy"
precision = 90 / (90 + 198)       = 31.25%
recall    = 90 / 100              = 90%

Read the last line of the figure again, because it is the whole lesson. A model that flags nothing at all — a single line of code returning false — scores 99% accuracy on this population. The vendor's 98% is worse than doing nothing, as measured by the metric they chose to report.

Meanwhile the operational reality is 288 flagged documents of which 198 are false alarms. If review takes 25 minutes per document, that is 120 hours per 10,000 documents — two-thirds of it (about 82 hours, the 198 false alarms) wasted — and reviewer trust in the queue erodes within weeks — the well-documented failure pattern in alert-driven work, where a two-thirds false rate teaches reviewers to skim.

Note the phrase "at your prevalence." Precision depends on the base rate, so a vendor measuring on a curated set with 20% violations will report precision far above what you will see at 1%. The confusion matrix must be computed on a population with your prevalence, or converted to it.

The base-rate question handles the metric. Provenance handles the number's origin, and it is asked the way foundation is laid for an exhibit: not "is this true" but "who made this, from what, and under what conditions."

• Who assembled the evaluation set, and when? A vendor-assembled set is not disqualifying, but it changes what the number means, and the assembly date matters for the next question.
• Could the evaluation data have appeared in training? This is benchmark contamination, and for anything built on a model pretrained on web-scale data it is the default hypothesis rather than an exotic risk. Public benchmarks are on the public internet. The check is a private evaluation set the vendor has never seen.
• Who chose the baseline, and would a competent adversary have chosen it? "Three times better than the legacy system" invites the question of whether the legacy system was a serious comparator or a decade-old rules engine nobody defends. The honest baseline is the best cheap alternative, not the incumbent.
• What was the metric before it was this one? Metrics get selected after results are seen. Asking which metrics were computed, in what order, is the ML equivalent of asking what else the expert wrote before this report.
• What is the failure distribution? Not the aggregate error rate — the shape. Which cases fail, and are they clustered in a segment that matters to you? A vendor who cannot describe their failure modes has not examined them.

Suppose every previous answer is honest. Provenance is clean, the baseline is serious, the metric is precision at a stated threshold, the holdout was untouched. The model is genuinely good — on the vendor's distribution.

Your documents are written by your people, in your jargon, under your retention policy, with your prevalence of the thing being detected. A model trained on financial services communications and evaluated on financial services communications tells you very little about a logistics company's Slack. The only claim that matters is the one measured on your data, and no amount of interrogation substitutes for measuring it.

So the interrogation ends in a pilot design, and the terms of the pilot must be fixed before results exist — which is the same discipline as module 2's test-set policy, applied to a commercial negotiation.

Each entry is symptom, mechanism, corrective — and a pointer back to the module that explains why.

Three of these have the same underlying shape, and noticing it makes them easier to catch: the demo-to-production leap, benchmark trust, and accuracy-on-imbalanced-classes are all cases of accepting a measurement taken under conditions that differ from the conditions you will operate in. The other three — fine-tuning as reflex, neural nets on small tables, size equals capability — share a different shape: choosing an instrument by its impressiveness rather than by its fit to the problem. Two failure families, six named forms.

Every failure so far is detectable before launch. This one is not, because it does not exist before launch.

A model is a static file. It does not degrade, corrode, or drift. The world drifts. Your product adds a segment, a competitor changes their pricing and your customers' language changes with it, a regulation shifts what counts as a violation, a new integration starts sending documents in a format that did not exist during training. The model keeps applying the regularities it compressed from data that no longer describes the present, and its error rate rises quietly.

What makes distribution shift operationally nasty is that nothing pages anyone. The service is healthy, latency is flat, no exception is thrown. Predictions are still returned, confidently, and they are wrong more often than they used to be. In most organizations the discovery mechanism is a downstream human noticing that the outputs have felt off for a while.

Monitoring for it does not require labels, which is the part teams get wrong. You cannot compute accuracy in production without ground truth, but you can watch the inputs, and inputs are free. Track the distribution of key features and the rate of new categories; for text, track the share of inputs whose embedding sits far from every training cluster. Track the model's own output distribution too — a violation detector whose flag rate drifts from 2.4% to 3.9% has told you something even if you cannot yet say what.

Then commit at launch to two things: a re-evaluation cadence (quarterly, with a freshly labeled sample), and a trigger threshold that pulls the re-evaluation forward. The cost of labeling 300 documents a quarter is small and known. The cost of finding out in month ten is neither.

The course reduces to three mechanisms and the questions they generate. Everything else was elaboration.

Three more questions come from the economics rather than the mechanisms, and they are the ones that most often change a decision: what does the simple version score? — the baseline demand that prices every claim. What does this cost per unit at our volume, and what does the cheaper option lose? — the escalation discipline from module 4's routing. And what would tell us this stopped working? — the monitoring commitment, made at launch or not at all.