语义搜索导论：为什么需要向量数据库 | Introduction to Semantic Search: Why We Need Vector Databases

第 1 章：语义搜索导论：为什么需要向量数据库

从一个常见的困惑开始

Consider this scenario: you're searching for "methods to make my computer run faster" on a search engine. What results would you expect? Most likely, the search engine would return articles about clearing disk space, upgrading RAM, disabling startup programs, or upgrading to an SSD. These results align with what you intended—they answer the core question embedded in your query. But what if you searched for "ways to speed up my morning routine"? A traditional keyword-based search engine might struggle here, because none of these words literally match anything related to computers. Yet semantically, both queries share a common pattern: they express a desire to improve efficiency through optimization.

This seemingly simple example reveals one of the fundamental challenges of information retrieval: human language is inherently ambiguous, context-dependent, and often expresses intent through indirect means. The words we use to describe what we're looking for rarely match the exact terminology in the documents that contain the answers we need.

This is precisely the problem that semantic search and vector databases are designed to solve. In this opening chapter, we'll explore why traditional search methods fall short, how semantic search fundamentally changes the game, and why vector databases have become an indispensable component of modern AI systems. We'll also map out the landscape of this technology ecosystem and establish a clear learning path for the journey ahead.

The Limitations of Traditional Keyword Search

How Keyword Search Works: A Brief Overview

To understand why semantic search represents a paradigm shift, we must first appreciate how traditional keyword search operates. The dominant algorithm for decades has been TF-IDF (Term Frequency-Inverse Document Frequency) and its more refined variant, BM25 (Best Matching 25).

TF-IDF, as the name suggests, calculates the importance of a term in a document based on two factors: how frequently the term appears in that document (term frequency) and how rare the term is across all documents in the corpus (inverse document frequency). The intuition is elegant—words that appear frequently in a specific document but rarely across the entire collection are likely to be semantically significant to that document.

BM25 improves upon TF-IDF by introducing a saturation function that prevents very common words in a document from being overweighted, and a document length normalization that accounts for variations in document size. Modern search engines like Elasticsearch and Solr still rely heavily on BM25 as their core ranking algorithm.

The Fundamental Problems with Pure Keyword Matching

Despite its effectiveness in many scenarios, keyword-based search suffers from several inherent limitations that become increasingly problematic as information volumes grow:

The Vocabulary Mismatch Problem represents perhaps the most significant challenge. Consider a medical researcher searching for literature on "heart attacks" but the relevant papers use the term "myocardial infarction." A pure keyword search would fail to retrieve these highly relevant documents because the exact strings don't match. This problem extends far beyond synonyms—in technical fields, different communities often use entirely different terminology for the same concepts. A software developer might search for "程序崩溃" (program crash in Chinese) while the relevant documentation uses "异常终止" (abnormal termination).

Lack of Semantic Understanding means that keyword search cannot distinguish between different meanings of the same word. If you search for "jaguar," do you mean the animal, the car brand, or perhaps an old IBM operating system? Traditional search engines treat all queries as bags of words, completely ignoring context and intent. The query "jaguar speed" might return articles about animal movement speeds or sports car performance depending on which interpretation matches the highest number of keyword occurrences.

No Concept of Semantic Similarity prevents keyword search from retrieving conceptually related content that doesn't share exact terminology. If you're researching "climate change mitigation strategies," you would ideally want documents about renewable energy, carbon taxes, reforestation, and electric vehicles—topics that are semantically related but might not contain the exact phrase "climate change mitigation." A keyword search would only find documents that explicitly use those specific terms.

Inability to Handle Complex Queries becomes apparent when users express nuanced information needs. Natural language queries like "What are some alternatives to Spotify that work offline and don't have ads?" or "Show me restaurants nearby that are quiet, romantic, and serve Italian food" are essentially impossible to answer correctly with pure keyword matching. These queries require understanding intent, constraints, and semantic relationships between concepts.

Quantifying the Vocabulary Gap

Research in information retrieval has quantified the vocabulary mismatch problem in concrete terms. A landmark study by Furnas et al. in 1987 found that when two people describe the same object, they use the same term only about 10-20% of the time. More recent research in the NLP community has confirmed that even professional indexers—the humans who carefully assign controlled vocabulary terms to documents—agree with each other only about 60-70% of the time.

This means that for any given concept, there are potentially dozens or hundreds of different ways to express it in natural language, and no single term dominates. A search system that relies purely on exact keyword matching can at best capture a small fraction of these expressions, leaving the vast majority of relevant documents hidden from users.

The Emergence of Semantic Search

Defining Semantic Search

Semantic search represents a fundamentally different approach to information retrieval—one that seeks to understand the *meaning* behind queries and documents rather than simply matching words. The term "semantic" derives from the Greek word "sēmantikos" (significant), and semantic search is indeed about capturing significance and meaning.

At its core, semantic search operates on a simple but powerful insight: if we can represent both queries and documents as points in a high-dimensional semantic space, then finding relevant documents becomes a matter of identifying points that are close to each other in that space. This geometric interpretation transforms the abstract problem of "understanding meaning" into the concrete mathematical problem of "measuring distance in vector space."

How Semantic Search Works: From Text to Vectors

The process of converting text into vectors suitable for semantic search involves several key steps, each of which we'll explore in detail throughout this book:

Tokenization breaks down raw text into individual tokens—typically words or subword units. Modern approaches often use subword tokenization (like Byte Pair Encoding or WordPiece) to handle out-of-vocabulary words and morphological variations more gracefully. For example, the word "running" might be tokenized as ["run", "##ning"], capturing both the root meaning and the grammatical modification.

Embedding Generation transforms these tokens into dense vector representations. Modern embedding models, typically based on transformer architectures like BERT, RoBERTa, or specialized models like SBERT (Sentence-BERT), are trained on massive corpora to capture semantic relationships. The training process ensures that semantically similar texts produce vectors that are close together in the embedding space, while semantically dissimilar texts produce vectors that are far apart.

The mathematical foundation of word embeddings traces back to the distributional hypothesis, famously articulated by lin