Finding Things

A cellar owner types barolo tannins into your wine app. Consider what a SQL engine does with that string and what a person means by it. The engine, given WHERE note LIKE '%barolo tannins%', has exactly one correct answer: the set of rows whose note contains that literal substring, in that order, with that spacing. In this corpus the answer is the empty set, and the engine is not wrong. It answered the question it was asked, perfectly.

The person meant something like: show me Barolo, or wines built like Barolo, and I care about structure — I am shopping for tannin, not for the word tannin. That intent is not recoverable from the string by any amount of parsing. It is recoverable, partially and probabilistically, from the string plus a model of how people write, a model of what is in the corpus, and a willingness to return things that only approximately qualify. That is the whole discipline.

So the useful frame is evidentiary rather than declarative. A query is not a specification; it is testimony about a need, given by a witness who is in a hurry, does not know your vocabulary, and will not be cross-examined. Your job is inference under ambiguity — deciding what the witness probably wanted and ordering the corpus by how well each document serves it. Relevance is the name for that fit, and unlike a predicate it is a judgment: it can be argued about, it varies by user and context, and it can only be settled by evidence about outcomes.

Suppose matching is solved. A user searches a 40,000-bottle catalogue for red wine and 3,000 documents qualify. Now what? The match set is a fact about the corpus and worth almost nothing to the user, who will look at somewhere between three and ten results and then either click or leave. The entire product — the thing they experience as "this search is good" or "this search is garbage" — is which three documents came first.

This is why the two-stage anatomy matters. Stage one, matching, is a cheap set operation that narrows a corpus to candidates. Stage two, ranking, spends the real effort deciding their order. Matching is a solved problem in the sense that a good data structure makes it fast and boring; ranking is where every remaining decision, every tuning knob, every regression, and every argument with your stakeholders lives.

Two systems can return the identical 200 documents and be a delight and a disaster respectively. That sentence is the whole argument, and it is worth saying out loud to any stakeholder who proposes measuring search by whether it "finds the thing" — it always finds the thing, somewhere on page nine.

Name the anti-pattern precisely so you can recognize it in a design doc. LIKE-as-search is shipping a substring predicate as the search feature of a product where users type freely. It fails in four structural ways, none of which is fixable by tuning the database.

It cannot rank. A predicate returns a set. Whatever order you see is the order the storage engine happened to produce, which means the first result is an accident. It cannot analyze. Substring matching has no concept of a word: %oak% matches "oaky" (good, by luck) and "Oakland" (bad, by the same luck), while %tannins% misses a note that says "tannin" and %rose% misses "Rosé" unless your collation happens to save you. It cannot tolerate error. barollo returns nothing, forever, with no signal to anyone that a customer just failed to find a $89 bottle. It cannot scale. A leading wildcard defeats a B-tree index, so every query is a full scan — and the moment somebody notices the latency, the team reaches for a database index, which fixes the one problem the users were not complaining about.

Be equally precise about when the predicate is right. If the field is an identifier, the user is an operator, the corpus is small, and the interaction is a lookup rather than a search — an admin panel that fetches a customer by exact account ID, a support tool that pulls a case number — then a predicate is not a degraded search engine, it is the correct tool. It is exact, auditable, has no index to fall out of date, and returns nothing when nothing matches, which is the honest answer. The mistake is not using LIKE; the mistake is using it where users type prose.

Here is the thesis the rest of the course elaborates. Between the words a person typed and the thing they were actually looking for there is a gap, and every mechanism in search is machinery for closing it. There are exactly two families of machinery, and they close it from opposite ends.

Keyword machinery closes the gap from the literal side. It forgives what you said: it decides what counts as a word, folds case and accents, strips plurals, tolerates a transposed letter, and weighs your terms so that a rare one counts for more than a common one. It is precise, cheap, inspectable, and completely blind to meaning — it cannot know that "like a Barolo but cheaper" describes a wine whose tasting note never uses the word Barolo. That machinery is m2 (the index), m3 (scoring), and m4 (the query side).

Vector machinery closes the gap from the meaning side. It matches what you meant: text becomes a point in a learned space where similar usage lands nearby, so a query and a document can match with no vocabulary in common at all. It is fuzzy about what you said, which is precisely the problem when what you said was a lot number. That machinery is m6, and its cost model is m6 as well.

Between the two bridges sits the thing that makes either of them improvable: evaluation (m5). Without a judged set of queries you can only argue about relevance, and the argument is won by whoever complained most recently. With one, relevance becomes a number with a denominator, changes become regressions or improvements, and "the search feels worse since Tuesday" becomes a testable claim. Then m7 joins the bridges — because the honest finding, once you measure, is that each engine visibly loses queries the other wins — and m8 puts the joined system under an AI feature, where retrieval quality becomes the hard ceiling on what any model can say.

Store the five cellar wines as rows and the natural question is document-shaped: for each row, does its note contain what I want? Answering it requires visiting every row, because a row store has no way to know which rows are candidates until it looks. Five rows is free; five million is a table scan you feel in the p99.

Invert it. Build a dictionary whose keys are terms and whose values are the documents containing them — the posting list for that term. Now the question is term-shaped: look up "firm", look up "tannin", intersect. You never touch a document that fails the query. The cost is two dictionary lookups plus a walk over two short lists, and — this is the property worth memorizing — that cost is a function of how many terms the user typed and how long those particular lists are, not of how many documents you own. Adding a million wines that do not mention firmness makes the query for firm tannins no slower at all.

Tokenization splits a stream of characters into the units you will index. It feels like a mechanical step and it is not: it is a policy decision that helps some queries and hurts others, and the corpus votes on which.

Splitting on whitespace and punctuation is the sane default for prose, and it immediately does something consequential to L-4471: the hyphen is punctuation, so the lot code becomes two tokens, l and 4471. Now a search for the exact lot code matches any document containing a stray "L" near a stray "4471", and — worse — the code no longer exists in the index as a single findable thing. Keep the hyphen and you have solved lot codes while breaking "American-oak aging" in d5, which now indexes the token american-oak and will not match a search for oak. Apostrophes have the same shape of problem ("producer's" as one token or two), and so do CJK scripts, where whitespace does not delimit words at all.

There is no policy that wins everywhere, which is why the working answer is not a better tokenizer but a second field. Index the tasting note with a prose analyzer and index the lot code as a single verbatim keyword — same document, two representations, each queried by the class of query it serves. That split is the corrective you will reach for repeatedly, and m4 shows what happens to the product when nobody makes it.

Tokens as typed are too literal to match reliably. Normalization flattens variants that carry no meaning difference: case folding so Barolo and barolo are one term, accent folding so Rosé and rose meet, punctuation stripping so tannins; loses its semicolon. Each fold buys recall and spends precision, and the spending is usually invisible — until a corpus of French producers discovers that accent folding has merged two distinct estates.

Stemming goes further, reducing inflected forms to a shared root so that a document saying "tannins" answers a query for "tannin". This corpus uses a light English stemmer — plural and common-suffix stripping, plus one hand-written rule mapping oaky to oak so that d3's "unapologetically oaky", d4's "no new oak", and d5's "American-oak aging" all land on the same term. That hand-written rule is worth noticing: it is the honest shape of production analysis, where a general algorithm gets you most of the way and a short list of domain rules covers the vocabulary that actually matters to your users.

The failure mode has two directions and they need different fixes. Over-stemming merges words that should stay distinct — an aggressive stemmer that maps both mature (d5, ready to drink) and maturation (a winemaking process) to matur will make a search for drink-ready wines return every wine whose note discusses barrel maturation, and precision for that query collapses. Under-stemming leaves forms unmerged, so rose petal never matches a note that says rose petals, and the miss is silent. The instinct after an over-stemming incident is to reach for a lighter stemmer everywhere, which trades one silent failure for the other; the better move is per-field analysis, keeping aggressive stemming on descriptive prose and light or no stemming on names and controlled vocabulary.

A posting list stores more than document ids. For each term it holds, per document, the term frequency (how many times the term occurs there) and the positions at which it occurs. Both are there to pay for something specific later: frequencies feed the scoring function in m3, and positions make phrase and proximity queries possible in m4. If you have ever wondered why an index is several times the size of the text it indexes, this is most of the answer.

Here is the whole cellar index, analyzed and inverted, with the query firm tannins traced through both structures.

Note the shape of the intersection: the engine walks the shortest list first. firm has one posting, so the candidate set is one document before tannin's three postings are even consulted. This is why a query containing one rare term is fast regardless of how common the other terms are, and it is the same fact — rarity is informative — that m3 turns into a scoring signal.

Now the constraint that governs the rest of the keyword half of this course. The analyzer ran once, when the document was written, and what it produced is all that exists. There is no copy of the original text inside the posting lists; there are terms. If the analyzer did not emit tannin for d1, then no query, however cleverly constructed, can find d1 by that word — the form was never indexed, and query time has nothing to search.

This makes the analyzer a contract with two signatories. The index side promises to have transformed documents a certain way. The query side must promise to transform queries the same way, or the two never meet — that is m4's mirror-or-miss rule, and it is the single most common silent search bug in production. And when you change the contract, you have not upgraded the index; you have created an index whose old documents were written under the old terms and whose new documents are written under the new ones. The corpus must be reprocessed from the source of truth.

The forward pointer is worth planting now. m3 will score using the frequencies stored here. m4 will match phrases using the positions stored here, and will show what happens when the query side breaks the mirror. m8 will note that this index is a derived copy of a source of truth and derived copies drift — which is the same fact as "the analyzer ran once," seen from operations rather than from correctness.

Term frequency. A document that says "tannin" four times is more likely to be about tannin than one that says it once. Without this signal, every document containing a query term scores identically, and ordering collapses back to whatever the storage engine felt like — the m1 failure. So: more occurrences, more score.

Term rarity. Not all query terms are equal evidence. In this five-wine corpus, tannin appears in three of five documents and firm in exactly one. A document matching firm has told you something; a document matching tannin has told you almost nothing, because most of the corpus would have. Rarity — the signal usually called inverse document frequency — makes a match on a discriminating term worth more than a match on a ubiquitous one. Without it, the query the 2016 barolo would be dominated by whatever document repeats the most, which is a real and famous failure.

Document length. Frequency alone rewards volume. A 4,000-word tasting essay will mention almost any wine term more often than a twelve-word note, not because it is more about that term but because it is longer. Length normalization divides out the advantage, asking not "how many times" but "how many times, for a document this size." Without it, your longest documents win every query, and the symptom is a results page where the same handful of verbose documents appear for unrelated searches.

Each signal exists to correct one error, and the corrections are independent — which is why you can reason about them separately when a ranking looks wrong. The diagnostic question is always: is this document winning on repetition, on rarity, or on brevity?

Term frequency helps, but taken literally it is exploitable and wrong. A note mentioning tannin ten times is not ten times more about tannin than one mentioning it once. The first occurrence carries enormous information — the document is about this at all. The second confirms it. The seventh tells you the author is repetitive.

Real scoring functions therefore bend the frequency curve so each additional occurrence contributes less than the one before it, approaching a ceiling. This is saturation, and it is the single most important behavioral property of BM25 — the thing that distinguishes it from the naive count-based scoring that preceded it. The formula's constants exist to control how sharply the curve bends; the behavior is what you need.

Take the canonical query firm tannins against the three documents that match either term after analysis — d1, d2, and d3 — and score them by hand. The model below is deliberately simplified: per-term contribution is saturated frequency × rarity weight, and the document's total is adjusted for length. It is BM25-shaped illustrative arithmetic, not BM25's formula; it preserves the behavior you need to reason with and drops the constants you would never tune by hand anyway.

The rarity weights come from the corpus: firm appears in one document of five, so it is highly discriminating (weight 3.0); tannin appears in three of five, so it barely narrows anything (weight 1.0). The length adjustment is the corpus average note length, 10.8 tokens, divided by the document's length — d1 is 10 tokens (1.1×), d2 is 15 (0.7×), d3 is 8 (1.35×). Every term here occurs once in every document, so saturated frequency is 1.0 throughout and this comparison is decided entirely by rarity and length.

Two habits come out of this. First, when a ranking surprises you, decompose it: most search engines will show you the per-term contributions on request, and the answer is nearly always that one document matched a rarer term or is much shorter than you assumed. Second, notice how little work term frequency did here. In short documents — product titles, tasting notes, support ticket subjects, control descriptions — almost every term occurs once, so rarity and length decide everything, and tuning frequency behavior is theatre.

One thing the anatomy above ignores: where the match happened. A wine whose name is "2016 Barolo Monvigliero" and a wine whose tasting note happens to say "unlike the Barolo next to it" both match barolo, and they are not remotely equally relevant. Field weighting fixes this by multiplying a field's contributions before they are summed — a name match at 3×, a grape or region match at 2×, a note match at 1×.

The justification has to come from intent, not from taste. Why does the name deserve a boost? Because a name is what a document is, while a note is what someone said about it, and a user typing a proper noun is far more often naming the thing than describing it. That is an argument you can defend, and — crucially — an argument that could be wrong for a different corpus. In a support-ticket search, the title field is written hastily and the body carries the diagnosis, so the same reasoning can point the other way.

Here is generation 1 of the wine cellar's tuning pass, run honestly. The complaint was that searching a producer name surfaced other wines' notes first. The fix — boost name to 3× — worked: the producer query snapped into place. The regression, discovered later, was that nebbiolo now returned d2 ("2021 Langhe Nebbiolo", name match, 3×) far above d1 (2016 Barolo Monvigliero, whose grape is Nebbiolo but whose name does not say so), and a customer shopping the grape got the cheap bottle and not the serious one. One query improved, one regressed, and the team had no way to know which effect was larger.

A boolean query is a specification. (nebbiolo OR barolo) AND tannin NOT oak defines a set exactly, and the engine's job is set arithmetic. It is precise, reproducible, auditable, and completely unforgiving: if the corpus says "Nebbiolo d'Alba" and you asked for nebbiolo, you get it; if it says "Nebbiolo-based", your tokenizer had better agree with you. And it typically arrives unranked, because a set has no order — which is fine, since the user who wrote that query has accepted the burden of specifying exactly what they want.

A free-text query is testimony, in m1's sense. The engine treats terms as evidence, matches loosely, scores, and orders. It forgives the user's vocabulary and spelling and returns something useful for almost any input. What it gives up is determinism: you cannot fully predict the result set from the query, two similar queries can behave differently, and "why did this document appear" requires an explain plan rather than a reading of the query.

The correct product answer is almost never either-only. Ship free text as the default surface and expose structure — filters, facets, an advanced mode, quoted phrases — for the users who want to bear the burden. What is genuinely wrong is forcing boolean syntax on a consumer product, and what is quietly wrong is offering only free text in a compliance or research context where someone will eventually have to defend the completeness of a search.

Bag-of-words matching loses word order the moment the document is indexed. That is usually fine and occasionally disastrous: "firm tannins" and "tannins, though not firm" contain identical terms, and only the stored positions from m2 can tell them apart.

A phrase query consumes those positions directly. Look up firm (d1, position 4) and tannin (d1, position 5; d2, position 6; d3, position 6), then keep only documents where the positions are consecutive in the right order — d1 survives, the others were never adjacent. Proximity search softens the constraint: instead of requiring distance 1, score by distance, so terms three words apart contribute more than terms thirty apart. This is what /s was doing in Westlaw, and it is the better default for most prose search because it captures the intuition — related words cluster — without the brittleness.

Phrase strictness helps where the phrase is a name or a quotation: "Gran Reserva" is a legal designation, not two adjectives, and matching it loosely returns every Spanish wine with a long word next to a reserva. It strangles recall where users are describing rather than naming, because a two-word phrase must appear in exactly that order to survive, and prose does not cooperate. The workable pattern is to run both — the phrase interpretation as a scoring boost rather than a filter — so documents containing the exact phrase rank first while documents containing the terms scattered still appear below.

Fuzziness matches terms within a small edit distance of what was typed: barollo reaches barolo at distance 1 (one deletion), nebiolo reaches nebbiolo at distance 1. The cost is that everything within that radius now matches, so the term stops being one term and becomes a neighborhood, and precision falls by however many real words happen to live nearby.

On prose, the trade is usually good and the standard configuration reflects it: distance 1 for medium words, distance 2 for long ones, exact matching for short words where a single edit reaches too many unrelated terms. On identifiers it is not a trade, it is a defect. L-4471 and L-4477 are at edit distance 1 and refer to different lots of different wine at different prices. A fuzzy match between them does not help a confused user; it silently ships the wrong bottle and produces an inventory discrepancy that nobody traces back to a search setting three months later.

The general rule is worth stating as a principle, because it recurs: forgiveness is appropriate where being approximately right is useful, and inappropriate where being approximately right is being wrong. Prose is the first case. Codes, names, quantities, and dates are the second.

Now the failure m2 set up. The analyzer is a contract with two signatories, and the query side is the one that breaks it — usually by accident, usually during a migration, always silently.

Suppose the index stems and the query pipeline does not. A user types tannins. The query side passes tannins to the lookup. There is no posting list for tannins; the term was stemmed to tannin at write time, and tannin's list — containing d1, d2, and d3 — sits in the same dictionary, inches away, unreachable. The engine returns zero results and reports success, because zero results is a valid answer to a well-formed query. No exception is raised. No log line is written. Both halves of the system are working exactly as configured.

Notice the shape of the diagnosis, because it generalizes: when a failure sorts by form of the word, look at analysis; when it sorts by age of the document, look at reindexing (m2); when it sorts by topic, look at ranking (m3). Three symptom patterns, three different stages, and the misdiagnosis in each direction wastes a week.

One more query class deserves a flag before it becomes m6's problem. Try red with no oak against the fuller catalog you will judge in m5 — where red and no are ordinary common words, not rare tokens that happen to single out one bottle. The analyzer produces red, with, no, oak; of these only oak carries real weight, and it points at the oakiest wines; so the two documents that are most about oak ("unapologetically oaky", "long American-oak aging") score highest, because term matching has no operator for negation inside free text — no is just another token, powerless to subtract oak from the ranking. The correct answer ("no new oak") is buried beneath the very wines the user ruled out. No query-side tuning fixes this: fuzziness does not help, phrase matching does not help, field weighting does not help. Hold that failure; it will still be there in m6, and it will take until m7 to resolve.

Precision asks: of the results you returned, what fraction were relevant? It is the junk metric — low precision means the user is reading through noise. Recall asks: of the documents that were relevant, what fraction did you return? It is the missing-treasure metric — low recall means the right document exists in your corpus and the user never saw it.

The asymmetry that matters is in who reports them. Bad precision is visible: the user sees the junk and complains about it. Bad recall is invisible: nobody can report a result they were never shown. A search system with terrible recall generates almost no complaints — it generates silence, abandonment, and a support ticket six months later that says "I thought we didn't carry that." This is why a team optimizing on complaint volume drifts steadily toward high precision and unmeasured recall, and why the metric you are not tracking is usually the one degrading.

The two trade against each other by construction. Loosen matching to catch d2 for the query like a Barolo but cheaper — add synonyms, drop a required term, expand fuzziness — and recall rises while a crowd of marginal documents arrives with it, lowering precision. Tighten to remove the junk and the treasure goes back into the dark. There is no configuration that maximizes both, which is why the real question is never "is our search good" but "which error can we afford here."

The instrument is unglamorous: a list of real queries, each with human judgments about which documents should have come back. Sample queries from your logs rather than inventing them — invented queries encode what you think users type, which is the belief you are trying to test. Weight toward head queries for coverage and include a deliberate tail sample, because the tail is where semantic failures hide.

Here is the wine cellar's set: ten queries over a twelve-document corpus, with generation-1 keyword search measured against it. Binary judgments (relevant or not) are enough to start; graded judgments come later if you need them.

The arithmetic is deliberately checkable. Across the eight free-text queries there are 13 hits out of 24 slots, so mean precision@3 is 13/24 = 0.54. Counting relevant documents recovered anywhere in the top ten gives 19 of 24, so mean recall@10 is 0.79. The two identifier queries are reported separately as found-at-1: 2 of 2. That separation is not fussiness — a query with exactly one relevant document can never exceed 0.33 on precision@3, so averaging it in would drag the number down for a reason that has nothing to do with quality. Choosing a metric that can actually be achieved is part of building the instrument.

Two things about maintenance. First, judgments are per result, not per query: "J3 was bad" is an opinion again, while "for J3, d1 is not relevant and d2 is" is data. Second, the set is a living artifact — the corpus grows, query patterns shift, and a set built at launch and never touched becomes an instrument measuring a system that no longer exists.

If measuring search quality feels like engineering pedantry, consider that your first career already litigated it — under oath, with sanctions attached.

E-discovery is a recall problem with a legal standard. A producing party must find the responsive documents in a corpus that may run to millions, and "we searched and found these" is not a defense on its own; the adequacy of the search itself is discoverable and arguable. Technology-assisted review made this explicit: parties negotiate the protocol, sample the null set to estimate what the process missed, and report recall as a number that the other side is entitled to challenge. Courts have engaged directly with whether a given recall level was reasonable. Nobody in that room thinks recall is a metric an engineer invented to look busy — it is the quantity that determines whether the production was adequate.

Take two things from this. First, the discipline transfers: the null-set sample is a judged query set, built to estimate the bottom-left cell of Figure 5.1 precisely because that cell is invisible by construction. Second, the metric choice is a consequence of asymmetric cost, and that reasoning generalizes to your GRC work directly. When a compliance analyst searches the control library for evidence relating to an audit finding, a missed document can mean an unremediated gap presented as remediated. When a security engineer searches for every service touching a vulnerable dependency, a missed service is an unpatched service. In both cases the cost of a miss dwarfs the cost of reading three irrelevant results, so recall is the metric and precision is a comfort. Flip the asymmetry — a shopper browsing wine, where a miss costs a scroll and junk costs the sale — and precision leads.

The judged set is not a launch artifact. It is a regression suite, and it belongs in the same mental slot as your test suite: every relevance change — an analyzer edit, a field boost, a new synonym list, a swapped engine — runs against it before it ships, and the output is a diff.

That diff changes the conversation in a specific way. "The director's query looks better" becomes "mean precision@3 moves from 0.54 to 0.58, J2 and J7 regress by one hit each, and here is what regressed and why." You may still ship it. The point is not that the number decides; the point is that you now know what you are trading, the trade is written down, and in six months somebody can find out what the system used to do.

The generation-1 baseline is now established and binding: mean precision@3 0.54, mean recall@10 0.79, identifier lookups 2/2. Look at where it fails. J3 scores zero and J4 scores 0.33, and those two share a signature — the relevant documents share little or no vocabulary with the query, or the query contains a structural relationship (negation) that terms cannot express. That is not a tuning failure, it is the ceiling of the machinery, and the number now exists to prove it. m6 attacks exactly that hole.

An embedding is a learned function from text to a fixed-length list of numbers — a point in a space of several hundred to a few thousand dimensions. The function is trained on an enormous amount of text with an objective that amounts to: put passages that get used in similar contexts near each other, and push dissimilar ones apart. Whatever ends up encoded in those coordinates is whatever served that objective.

That last sentence is the honest description, and it is worth resisting the more flattering ones. The vector does not represent truth, or logic, or the properties of the wine. It represents usage patterns in the training data — a compressed prior about which texts behave like which other texts. It is extremely good at that, and the consequences are useful: "structured Piedmontese red" and "tar and roses, firm tannins" land near each other despite sharing zero words, because in the training corpus those phrases occur in the same neighborhoods of discourse. It is also the reason the failures in section three are not bugs to be fixed by a better model — they are what you get when your retrieval signal is discourse similarity.

If documents are points, similarity is a geometric relation. The standard measure is cosine similarity: the cosine of the angle between two vectors, running from 1 (same direction) through 0 (unrelated) to −1 (opposite). Angle rather than distance, because magnitude in these spaces tends to encode uninteresting things like text length, and you want a measure that ignores it.

Retrieval then has a pleasingly simple shape: embed the query with the same model that embedded the documents — this is the vector world's mirror-or-miss rule, and mixing embedding models produces coordinates in unrelated spaces and results that look like a random shuffle — then find the document points closest to the query point.

Read the map twice. The first reading is the promise: Q-C, the query that scored zero in the judged set, lands next to exactly the right document, and it does so with no shared vocabulary. The second reading is the warning, and it is the same picture: Q-E is a query with a precise, checkable answer — the user said "the 2016 Barolo" — and the nearest point is the wrong wine, returned with a high similarity score and no indication that anything went wrong.

Four failure classes, each with a geometric reason and each visible in this corpus.

Exact identifiers. Query L-4471. The string carries no usage pattern — it is an arbitrary label, and whatever position the model assigns it is essentially arbitrary too, driven by superficial character similarity to other codes. The nearest neighbors will be other lot codes and whatever else lives in that corner of the space, none of them meaningfully related. Keyword search answers this in one exact lookup; the vector system answers it with confident garbage.

Proper names with no training footprint. A small producer whose name appeared nowhere in the training data has no learned position. The model still produces a vector — it always produces a vector — assembled from subword fragments, which is to say from how the name is spelled rather than from anything about the producer. Well-known names work beautifully; obscure ones fail silently, and your catalogue is mostly obscure ones.

Negation and structure. Query red with no oak. The words "no" and "oak" appearing together produce a vector that sits near the oak region, because the passage is discussing oak. Geometry has no NOT operator: there is no direction in the space that means "exclude the neighborhood you are pointing at." This is why J4 fails on both engines, and it is a property of representing meaning as position, not a deficiency of any particular model. The same applies to quantities and comparatives — "under $40", "older than the 2015" — which are structural constraints wearing prose.

Out-of-domain queries. Jargon the model never encountered maps to some position anyway, and that position returns neighbors with respectable-looking scores. In a specialized corpus — internal control identifiers, proprietary product codenames, a house vocabulary — this is the common case rather than the edge case.

The naive way to find the nearest vectors is to compare the query against every document vector and keep the best. It is exact, trivially correct, and O(corpus) per query — the full scan that the inverted index was invented to eliminate, reappearing in the vector world wearing different clothes.

Do the arithmetic before dismissing it. A million documents at 768 dimensions is 768 million multiply-adds per query, plus the memory bandwidth to stream three gigabytes of float32 vectors past the processor. On modern hardware with good libraries that is tens to low hundreds of milliseconds for a single query, and it does not amortize across concurrent users — every query pays it again. At ten thousand documents brute force is genuinely fine and you should not build anything more complicated. At ten million it is not a latency problem, it is an architecture problem.

Note that the inverted index escaped this trap structurally: it never examines a document that lacks a query term, so most of the corpus is untouched by construction. Vector search has no equivalent shortcut, because every document has some distance to the query — there is no such thing as a document that does not contain the query point. Nearness is not a filter. That is why approximation arrives here not as an optimization but as the price of admission.

An approximate nearest neighbor index pre-organizes the vectors so a query can visit a small fraction of them. The families differ in mechanism — partition the space into cells and probe only nearby cells; build a navigable graph and greedily walk toward the query; compress vectors so more of them fit in fast memory — but they share one behavior, and the behavior is what you need.

They sometimes miss the true nearest neighbor. Not through a bug: by design. The search visits some regions and not others, and a genuinely close document sitting in an unvisited region is simply not found. The measure of this is recall@k — of the true top-k neighbors, what fraction did the index actually return — and every ANN index runs below 1.0. A typical production default sits somewhere near 0.95, which sounds like a rounding error and is not.

Two facts make this worth your attention as a product decision rather than an infrastructure detail. First, the trade is yours to make: the same index will run at 0.78 or 0.99 recall depending on a parameter, and the right point depends on whether a missed document costs a scroll or an audit finding — the m5 reasoning, applied to a config value. Second, ANN recall compounds with everything else. If retrieval feeds a RAG feature (m8), a document the index never returns is a fact the model never sees, and the resulting wrong answer will be diagnosed as a model problem by everyone who does not know this parameter exists.

Same judged set, same corpus, three canonical queries drawn out for inspection, and both engines' full numbers reported.

Now the aggregate, on the same ten-query judged set from m5, with the identifier queries reported separately as before.

Read the middle row carefully, because it is the row that gets misread in migration proposals. Vector-only improves both relevance averages substantially — and takes identifier lookups from perfect to zero. If your dashboard is mean precision, generation 2 is a triumph; if your warehouse team searches by lot code all day, generation 2 is an outage that does not page anyone. The average concealed a categorical regression, which is the general hazard of steering on a single quality number.

You now have two ranked lists per query and need one. The obvious move — average the scores — is numerology. A BM25-family score is an unbounded sum of term contributions that lands somewhere around 2 to 15 in this corpus and shifts with corpus statistics and query length. A cosine similarity is bounded in [−1, 1] and, for a real embedding model, clusters tightly in a narrow band like 0.65 to 0.95. Add or average those and the keyword branch dominates every fused result by sheer magnitude, and no amount of hand-tuned weighting fixes it durably, because the keyword scale moves as the corpus grows while the cosine scale does not.

Reciprocal rank fusion sidesteps the problem by discarding the scores entirely and keeping only what is comparable across engines: position. Each document scores 1/(k + rank) in each list it appears in, summed across lists, with k a smoothing constant conventionally set to 60 to keep the top few positions from dominating absolutely. Rank one contributes 1/61 ≈ 0.0164; rank two, 1/62 ≈ 0.0161; rank three, 1/63 ≈ 0.0159. The differences within a list are tiny — which is the design intent. What moves a document decisively is appearing in both lists.

Work it on J5, austere age-worthy piedmont red. Keyword returns [d1, d2, d4]; vector returns [d2, d11, d1].

The structure is visible in the arithmetic: documents endorsed by both engines score roughly twice what single-engine documents score, so agreement is the dominant signal and neither branch's internal confidence can bully the result. That is the property you want — not because both engines are equally trustworthy, but because their errors are largely uncorrelated, and a document both a lexical and a semantic matcher independently liked is a genuinely stronger candidate than one either liked alone.

Fusion has a limit, and Q-C shows it exactly. Keyword returns [d1, d6, d3]; vector returns [d2, d11, d7]; the lists are disjoint, so d1 and d2 both score 1/61 and tie at the top. Fusion has said the honest thing — two engines each endorse their own leader, and I have no basis to choose — and it is right to say it. Breaking that tie requires reading the query and the documents together, which is the next stage.

Both retrieval engines share a structural limitation: they compare a query representation to a document representation that was computed before the query existed. A posting list was built at index time; a document vector was computed at index time. Neither has ever considered this query and this document together.

A reranker removes that limitation for a small number of candidates. It takes the pair — the full query text and the full document text — and scores them jointly, which lets it attend to relationships that no precomputed representation can hold. That is what resolves Q-D: given "red with no oak" and d4's note "Volcanic and taut; red fruit, no new oak, Burgundian in spirit", a model reading both together can register that the document asserts the absence the query requires, while d3's "unapologetically oaky" asserts the opposite. Negation stops being a geometric impossibility and becomes a reading-comprehension task, which is a thing models are good at.

The cost is the whole design constraint. Where retrieval is one index lookup per query regardless of corpus size, reranking is one model inference per candidate. Reranking 40 candidates is 40 inferences; reranking 10,000 is 10,000, and no product's latency budget survives that. So the architecture is a funnel — cheap and wide first, expensive and narrow last — and the candidate count is a line item in a latency budget rather than a free parameter.

Choose the depth by measuring, not by taste. Run the judged set at rerank depths of 10, 20, 40, and 100 and plot quality against latency; in most corpora the curve flattens well before 100, because a document ranked eightieth by two independent retrievers is rarely the answer. Depth 40 in the cellar's case costs 180 ms and captures essentially all the available gain.

Hybrid is not a default to adopt unexamined; it is complexity that has to earn its place. Three honest positions.

Keyword-only is right for small structured corpora, identifier-heavy lookup, and audit-demanding domains. A 3,000-document policy library searched by title and control ID has no semantic gap worth an embedding pipeline: the vocabulary is controlled, the users are experts, and the ability to explain exactly why a document matched is a feature that compliance will ask for. It is also dramatically cheaper — no embedding inference at index time, no vector store, no second index to keep fresh.

Vector-only is defensible when queries are uniformly semantic and identifiers genuinely do not exist in the domain: a support-article search where users describe symptoms in their own words and never type a product code, an internal knowledge base whose documents have no stable names. Verify the premise before you rely on it, because "our users never search by identifier" is a claim your logs can test and is wrong more often than teams expect.

Hybrid earns its complexity when the query mix spans both classes — which product search and RAG retrieval almost always do, because catalogues have names and codes and users also describe what they want.

Generation 3, measured on the unchanged judged set, closes the arc: mean precision@3 0.92 against generation 1's 0.54 and generation 2's 0.79; mean recall@10 0.96; identifier lookups back to 2/2. Both single-engine baselines are beaten on both relevance metrics, and the categorical capability is preserved rather than traded. That last clause is the one to carry into a design review: the win is not that the average went up, it is that no query class got worse.

Draw the pipeline and the constraint is obvious. Documents are chunked, chunks are indexed, a question retrieves some of them, the retrieved text becomes context, and a model writes an answer from that context. The model's input is the retrieved passages. If the fact the user needs is not in them, the model has three options: say it does not know, decline, or invent something plausible. Under commercial pressure to be helpful, the third happens more than anyone would like — and the output is fluent, confident, and wrong, which is the most dangerous class of answer a system can emit, because it carries no signal that anything failed.

So retrieval sets a ceiling. A perfect generator over bad retrieval produces confident nonsense; a mediocre generator over excellent retrieval produces something merely clumsy and correct. This is why the phrase "we need a better model" is usually the wrong first move, and why every skill in modules 1 through 7 is now load-bearing under an AI feature: the analyzer, the scoring function, the judged set, the ANN recall parameter, the reranker. Everything upstream of the model is the R.

Documents are usually too long to retrieve whole, so they are cut into chunks, and the chunk is what gets indexed, retrieved, and handed to the model. Which means the chunk boundary decides what facts exist as retrievable units. That is an index-time contract in exactly m2's sense: decided once at write time, unchangeable at query time, and requiring a reindex to alter.

The size trade is real in both directions. Chunks that are too small fragment context — a passage saying "it needs another decade" is useless without whatever "it" refers to. Chunks that are too large dilute: a 3,000-word chunk embeds to a vector that averages a dozen topics and is strongly similar to nothing, and in the keyword branch its length normalization penalty (m3) buries it. Somewhere in the middle is right for your corpus, and the way to find it is to run the judged set at several chunk sizes rather than to adopt a default.

The boundary policy matters more than the size. A fixed character count cuts wherever the counter lands, which is routinely mid-sentence, mid-clause, and mid-fact.

Structure-aware chunking is more valuable the more structured the corpus is, and it is worth naming what "structure" means in a legal or compliance document: a contract's retrievable unit is the clause, not 512 characters. A control catalogue's unit is the control. Cutting a clause in half does not merely lose text — it can invert its meaning, because the exceptions and carve-outs live at the end.

Here is the cellar assistant's incident in full, because the sequence of moves is the transferable part.

The report. A customer asks the assistant, "When should I drink the Barolo Monvigliero?" It answers: "It is drinking well now — the tannins are firm and the wine is showing its austere side, so it is ready to enjoy." That is wrong, and confidently so. The note says the opposite: austere now, needs a decade.

Move one: read the retrieved context, not the prompt. The instinct is to open the system prompt and start adding instructions about carefulness. Resist it. Pull the passages the retriever actually returned for this question and read them as the model saw them. Doing that here surfaces the answer in about four minutes: the top retrieved chunk ends with the string ...austere now, needs. It just stops. The model did not hallucinate; it read a truncated claim and completed it in the most natural direction.

Move two: locate the earliest failing stage. Working backward through Figure 8.1: generation is faithful to its context, so not there. Context assembly passed through what retrieval gave it, so not there. Retrieval returned the most relevant chunk that existed — so not a ranking failure either. The failure is at chunking: a fixed 150-character window cut the drink-window claim in two, and the fragment carrying "a decade" was a ten-character orphan that will never win a retrieval for anything.

Move three: fix at the correct stage. Switch to sentence-aware chunking with a size ceiling and one sentence of overlap, reindex the corpus, and re-run the judged set to confirm nothing else moved. No prompt was edited and no model was changed — worth stating plainly in the incident write-up, because it is the sentence that changes how the next incident gets triaged.

Move four: prevent recurrence. Add this question and its correct answer to the retrieval judged set as a regression case. A fix that is not in the regression suite is a fix with a shelf life.

The index is a derived copy of a source of truth, and derived copies drift. This is the m1 framing coming due: the database is the system of record and the index is a query-optimized projection, which means something must continuously reconcile them.

Two shapes. A full reindex rebuilds everything from source — simple, self-correcting, and expensive enough that you run it on a schedule rather than continuously. Incremental indexing consumes change events and updates affected documents — cheap per change, near-real-time, and subtly fragile, because it accumulates whatever the event stream drops. Most production systems run both: incremental for freshness, periodic full reindex as the correctness backstop. Note that an analyzer or chunking change forces a full reindex regardless (m2), so the machinery has to exist anyway.

State freshness as a contract with a number in it: search reflects catalogue changes within five minutes at p99. A number can be monitored and can be wrong; "near real-time" cannot be either.

The failure that matters is the silent one. An indexer stops consuming — a poison message, an expired credential, a deploy that quietly fails its health check — and nothing user-facing errors. Search stays fast. Results keep arriving. They are simply the results from whenever the pipeline stopped, and the divergence grows every hour. Users experience it as "I added that bottle yesterday and it isn't there" and "this one sold out last week and it's still showing," which sound like two unrelated data bugs and are one stalled consumer.

Of everything a search product can instrument, the highest-signal stream is the list of queries that returned nothing. Each one is a user telling you, in their own words, exactly what they wanted and did not get — unprompted, specific, and free. There is no research method that produces better material, and most teams do not log it.

Mined weekly, zero-result queries sort into three bins, and the sorting is the work.

Vocabulary gaps. Users say something your corpus does not. Customers search claret and your notes say Bordeaux; they search orange wine and your notes say skin-contact. The corrective is a synonym list or a query-expansion rule, and the discipline is that each addition is a relevance change and runs against the judged set like any other.

Content gaps. Users want something you do not have. Forty searches for non-alcoholic in a month with zero results is not a search bug; it is a merchandising finding delivered by your search box, and it belongs in a product review rather than a bug queue. This is the bin that pays for the instrumentation.

Pipeline bugs. The m4 mismatch class shows up here first and unmistakably: a cluster of zero-result queries sharing a morphological signature — all plurals, all accented, all uppercase — is a broken analyzer contract announcing itself. Because this signature is so distinctive, a weekly scan of the zero-results report is effectively a monitor for a bug class that produces no errors anywhere else.

Note the loop this closes. The zero-results report supplies real queries; those queries populate the judged set (m5); the judged set referees the tuning (m3, m4, m6, m7); the tuning changes what floats to the top — which was m1's claim about what the product actually is. Search is not a feature you ship; it is a loop you run, and the report is where the loop takes its input from reality.

One last framing worth carrying out of this course. Every module named a mechanism and its failure mode, and every failure mode had the same property: it is silent. A mismatched analyzer returns zero results and reports success. An ANN index drops documents without mentioning it. A vector store answers an identifier query with confident garbage. A stalled indexer serves stale data at full speed. A chunk cut mid-sentence produces a fluent, wrong answer. Nothing in a search stack raises an exception when it stops being useful — which is why the instrumentation in this module and the measurement in m5 are not operational hygiene layered on top of the real work. They are the only reason you would ever find out.