The RAG Architecture: Beyond the Simple Retrieval-Generation Pipeline

The RAG Architecture: Beyond the Simple Retrieval-Generation Pipeline

When a legal research assistant powered by a RAG system confidently cites a case that doesn't exist, or when a technical support chatbot retrieves documentation so misaligned with the user's actual problem that the generated response makes things worse, we are witnessing the consequences of treating RAG as a solved problem. The architecture—retrieve relevant documents, stuff them into a prompt, generate a response—appears deceptively simple. Yet production RAG systems fail in ways that are subtle, systemic, and often traceable to specific architectural decisions made during design rather than implementation. Understanding why these failures occur requires moving beyond the conceptual diagram of a retrieval pipeline and examining the formal properties of the conditional generation problem that RAG attempts to solve.

This chapter establishes the foundational architecture of RAG systems by first formalizing what RAG is mathematically: a conditional generation problem where the probability of a response depends jointly on the query and a knowledge base. From this formalization, we derive the three-stage pipeline and analyze where quality degrades at each stage. We then develop a taxonomy of failure modes that distinguishes between recall failures (missing relevant information), precision failures (retrieving the wrong information), coherence failures (failing to synthesize retrieved content properly), and hallucination amplification (the generation component filling gaps with plausible falsehoods). With this failure taxonomy established, we examine the architectural variants that production systems employ to mitigate different failure modes. The chapter concludes with a detailed case study dissecting a production RAG system deployed for legal document Q&A, examining how each component was engineered and what went wrong during initial deployment.

Formalizing RAG as a Conditional Generation Problem

The naive description of RAG—retrieve documents, generate response—obscures the probabilistic nature of what the system is actually computing. A RAG system is fundamentally computing a conditional probability distribution over responses, conditioned on both the user query and the contents of a knowledge base. Formally, we can express this as:

$$P(\text{response} \mid \text{query}, KB) = \sum_{D \subseteq KB} P(\text{response} \mid \text{query}, D) \cdot P(D \mid \text{query}, KB)$$

This formulation reveals two distinct computational problems that the system must solve. The first term, $P(\text{response} \mid \text{query}, D)$, is the conditional probability of generating a particular response given the query and a specific subset of documents from the knowledge base. This is the generation problem, and it is largely handled by the language model component. The second term, $P(D \mid \text{query}, KB)$, is the probability that a particular subset of documents $D$ is relevant to the query—this is the retrieval problem, and it determines which documents the system will ultimately use to ground the response.

In practice, production RAG systems do not compute the full probability distribution over all possible document subsets. The combinatorial explosion of subsets makes this computationally intractable for any non-trivial knowledge base. Instead, systems approximate this by retrieving a small number $k$ of documents and treating them as the relevant set. The approximation becomes:

$$P(\text{response} \mid \text{query}, KB) \approx P(\text{response} \mid \text{query}, D_{top-k})$$

where $D_{top-k}$ is the set of $k$ documents with highest retrieval scores given the query.

This approximation introduces the first critical design decision in RAG architecture: the value of $k$ and how documents are ranked. If $k$ is too small, the system may miss crucial information (recall failure). If $k$ is too large, the retrieved context becomes noisy, and the generation model may struggle to identify the most relevant information (precision failure) or may exceed context window limits. Furthermore, the ranking function that determines which documents enter $D_{top-k}$ is itself a source of quality degradation—if the ranker prioritizes documents that are superficially similar to the query rather than substantively relevant, the entire downstream generation suffers.

The formalization also clarifies why retrieval and generation cannot be optimized independently. The retrieval component determines the distribution of $D$ that the generation component must condition on, while the generation component's behavior depends critically on the quality of its conditioning context. A retrieval system that returns perfectly relevant documents is wasted if the generation model cannot properly interpret them, and a powerful generation model cannot compensate for retrieval that consistently returns irrelevant documents. This coupling between components is the fundamental challenge in RAG engineering.

The Three-Stage Pipeline: Where Quality Degrades

The standard RAG pipeline decomposes into three distinct stages: retrieval, augmentation, and generation. Each stage has specific failure modes, and understanding these requires examining the data transformations that occur at each step.

```mermaid flowchart TD A[User Query] --> B[Query Processing] B --> C[Retrieval Stage] C --> D{Retrieve Top-k Documents} D -->|k too small| E[Recall Failure] D -->|k too large| F[Precision Failure] D -->|Wrong ranking| G[Relevance Failure] C --> H[Augmentation Stage] H --> I[Context Assembly] I -->|Missing metadata| J[Citation Failure] I -->|Poor ordering| K[Coherence Failure] H --> L[Generation Stage] L --> M[Response Synthesis] M -->|Conflicting sources| N[Hallucination Risk] M -->|Gap filling| O[Fabrication] L --> P[Final Response] E --> P G --> P J --> P O --> P ```

Stage One: Retrieval

The retrieval stage transforms a natural language query into a set of document identifiers from the knowledge base. This transformation occurs through query encoding, similarity computation, and ranking. The quality of retrieval depends on three factors: the quality of the query representation, the quality of the document representations stored in the index, and the compatibility between query and document representation spaces.

Query processing often receives insufficient attention in RAG tutorials but can dramatically affect retrieval quality. A query like "What happened to the Thompson case?" might retrieve documents about the Thompson family, Thompson electronics, or the Thompson legal case depending on how the query is processed. Query processing may include named entity recognition to identify that "Thompson" refers to a case name, query expansion to include synonyms or related terms, or intent classification to disambiguate the query type.

The retrieval mechanism itself falls into two broad categories: sparse retrieval and dense retrieval. Sparse retrieval methods, exemplified by BM25 and its variants, represent documents and queries as sparse vectors over a vocabulary of terms. These methods excel at exact term matching and are robust to vocabulary mismatch between query and document. However, they struggle with semantic similarity that doesn't align with lexical overlap. Dense retrieval methods, exemplified by bi-encoder neural architectures like DPR or sentence transformers, embed documents and queries into dense vectors in a learned embedding space. These methods excel at capturing semantic similarity but may struggle with exact factual retrieval where surface forms matter.

The ranking function determines how retrieved documents are ordered. In simple systems, the ranking is the same as the similarity score used during retrieval. More sophisticated systems employ a two-stage retrieval architecture: an initial retrieval stage that efficiently retrie