The Rise of Semantic Search: Why Vectors Replaced Keywords

The story of search in the modern era is fundamentally a story about representation. For decades, information retrieval systems represented documents and queries as bags of words, and the resulting systems were fast, interpretable, and surprisingly effective for many tasks. But the world they were designed for — a world where query authors and document authors agree on vocabulary — has steadily eroded. Today's retrieval problems live in domains where users describe what they want in their own words, using synonyms, paraphrases, abbreviations, and frequently imprecise or incorrect terminology. The ticket in front of an IT support analyst, the legal brief referencing precedent, the patient query into a clinical knowledge base — all of these routinely involve vocabulary gaps that lexical search cannot bridge.

This chapter opens the book by establishing why semantic search exists at all. We begin with the concrete failure mode that keyword search cannot escape: the vocabulary mismatch problem. We then trace the intellectual evolution that produced the dense embedding paradigm, from latent semantic analysis to today's transformer encoders. We map out the architecture of a semantic retrieval system at a level of detail sufficient to ground every later chapter, and we close the chapter with a worked comparison: a small corpus of technical support tickets, indexed both lexically and semantically, retrieved against the same queries by both paradigms, with a side-by-side analysis of where each wins and why. By the end of the chapter you should be able to articulate, with specifics, when semantic search outperforms lexical search, what each component of a vector pipeline is responsible for, and what failure patterns tell you it is time to move beyond BM25.

1.1 The Vocabulary Mismatch Problem

To understand why semantic search is necessary, you have to understand precisely what lexical search cannot do. Lexical retrieval systems — and BM25 in particular — operate on a deceptively simple premise: relevance can be inferred from the overlap of distinctive terms between a query and a document. The mathematical machinery behind that premise is elegant, but the premise itself carries an assumption that is violated constantly in real corpora: that the person searching and the person who wrote the document used the same words.

1.1.1 How BM25 Actually Scores

Before diagnosing the failure mode, it is worth being explicit about the scoring logic. BM25 builds on term frequency (TF) and inverse document frequency (IDF), but introduces two saturation parameters that prevent very long documents from dominating. For a query $Q$ with terms $q_1, q_2, \ldots, q_n$, the score of a document $D$ is:

$$ \text{score}(D, Q) = \sum_{i=1}^{n} \text{IDF}(q_i) \cdot \frac{f(q_i, D) \cdot (k_1 + 1)}{f(q_i, D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)} $$

where $f(q_i, D)$ is the count of term $q_i$ in $D$, $|D|$ is the document length, $\text{avgdl}$ is the average document length across the collection, and $k_1$ and $b$ are tunable parameters (typical defaults are $k_1 = 1.5$, $b = 0.75$). The IDF component is:

$$ \text{IDF}(q_i) = \log \frac{N - n(q_i) + 0.5}{n(q_i) + 0.5} $$

with $N$ being the total number of documents and $n(q_i)$ the number of documents containing $q_i$.

Three things to notice. First, BM25 — like all lexical retrievers — treats each term independently. There is no notion that "car" and "vehicle" refer to the same concept; if the term is not present, it contributes zero to the score. Second, BM25 is fundamentally a bag-of-words scorer. Token order, syntactic structure, and semantic proximity are all discarded at indexing time. Third, IDF rewards rarity, which means a query term that does not appear in the document cannot contribute any relevance signal, even if the document is conceptually a perfect match. These three properties together produce the vocabulary mismatch problem.

1.1.2 A Concrete Failure: Synonymy and Paraphrase

Suppose you maintain a knowledge base of technical support articles. One article is titled "How to reset your password" and its body reads:

> "If you have forgotten your account password, navigate to the login page and click 'Forgot password'. You will receive an email with instructions to create a new password."

A second article reads:

> "Customers who are unable to authenticate may regain access by initiating a credential recovery flow from the sign-in screen. A reset link is delivered to the registered email address."

Now consider a user who types the query "I can't log in to my account." Under BM25, the tokens that survive normalization are typically something like `["cant", "log", "account"]`. Article one contains `log`, `account`, and arguably `password` (related but not present in the query). Article two contains none of `cant`, `log`, or `account` — it has `sign-in`, `credential`, `recover`, `access`. The score for article two will be near zero, and the user will not see the article that is in fact a perfect match for their problem.

This is the canonical failure of lexical retrieval. It is not a bug in BM25; it is a direct consequence of the model's design. Lexical retrieval assumes that vocabulary is shared between query and document. When that assumption breaks — through synonyms, abbreviations, paraphrases, jargon translation, or non-native language use — BM25 produces no signal at all.

1.1.3 Polysemy, the Other Side of the Coin

If synonymy causes BM25 to miss relevant documents, polysemy causes it to retrieve irrelevant ones. Consider the word "bank". A document about river banks and a document about financial banks both contain the same token, and BM25 cannot distinguish them based on context. Lexical search has no mechanism to know that "bank account" and "river bank" refer to entirely different concepts.

For some queries this matters less than for others. A query like "river bank erosion" is unambiguous in context. But for short, ambiguous queries — and real queries are short, often averaging three to five words — polysemy is a constant source of precision loss.

1.1.4 Why Stemming and Query Expansion Don't Fix It

The standard countermeasures in lexical IR — stemming, lemmatization, stopword removal, controlled vocabularies, synonym dictionaries — are partial fixes. Porter stemming reduces `authentication` and `authenticate` to the same stem (`authent`), which helps for morphological variants. But stemming cannot bridge `login` to `sign-in`, or `crash` to `unresponsive`. Query expansion using WordNet or a domain thesaurus can inject synonyms, but the expansion is brittle: it does not know that `slow` in "my laptop is slow" is more closely related to `unresponsive` and `laggy` than to `deliberate`. And manual thesaurus maintenance does not scale; the moment your domain introduces a new term, the thesaurus is out of date.

In short, lexical retrieval lacks any mechanism to know that two distinct-looking strings are semantically close, or that the same string is semantically ambiguous. It treats language as a discrete, surface-level token stream, and the real world does not behave that way.

1.2 From Sparse Vectors to Dense Representations

The intellectual journey from lexical retrieval to semantic retrieval is not a sudden break. It is a forty-year progression in which each generation of representation learned a little more about meaning, and a little less about surface form.

1.2.1 The Sparse Era: TF-IDF and the Vector Space Model

The starting point is the vector space model proposed by Salton and colleagues in the 1970s. Every document and every query is represented as a vector in a high-dimensional space whose dimensions correspond to terms in the vocabulary. A corpus with 500,000 unique terms produces vectors of dimension 500,000. Most entries are zero — hence the term *sparse* — and the few non-zero entries are weighted by TF-IDF.

Geometrically, this repre