What Is RAG and Why Should We Build One?

What Is RAG and Why Should We Build One?

Meet Lexi, the Robot Who Knew Too Much and Too Little

Imagine a robot named Lexi sitting in a bright yellow classroom. Lexi is brilliant. Ask Lexi anything about the Roman Empire and she will tell you that Augustus Caesar became the first emperor in 27 BCE, that Roman legions carried curved rectangular shields called scuta, and that the Romans built roads so straight that engineers still copy their design today. Ask her about the fall of Constantinople, the structure of a Roman aqueduct, or the names of the seven kings of Rome, and she will answer in perfect paragraphs, with dates, with confidence, with the calm certainty of a museum tour guide who has rehearsed the script a thousand times.

But ask Lexi one simple question: "Is it raining outside right now?" and she freezes. Her eyes blink. A small gear inside her chest makes a sad *click*. She says, "I am unable to access real-time weather information, but historically, rain has been observed on approximately 35% of days in temperate regions during spring months." She is not lying. She is not even wrong, exactly. But she has completely missed the point.

Lexi has three problems, and they are the same three problems that every large language model has in the year 2025. The first problem is that her knowledge is frozen at the moment her training finished. If her training data ended on January 31, 2024, she has never seen February of 2024, let alone today. The second problem is that she sometimes invents facts that sound right but are not, a behavior AI scientists politely call *hallucination*. The third problem is that she has never read your family's photo album, your school's lunch menu, or the secret recipe your grandmother wrote on the back of an envelope. She knows everything the public internet taught her, and nothing that lives only inside your house, your company, or your classroom.

These three problems are why we build something called a Retrieval-Augmented Generation system, which everyone shortens to RAG. RAG is not a new kind of robot. It is a small, clever habit you teach Lexi: before answering a question, she must go to a library, find the most relevant pages, and read them first. Then she answers, but only using what she just read. Suddenly Lexi can talk about today's weather (because the library has today's forecast), stop hallucinating (because the pages tell her the truth), and discuss your grandmother's recipe (because the library has a copy of it).

This book is the story of how to build that library, and how to teach Lexi — or any other language model — to use it well.

The Three Big Problems with Standalone Language Models

To understand why RAG exists, we have to look carefully at what large language models actually are. A large language model, or LLM, is a very large mathematical function. It is a file filled with billions of numbers called *parameters*. When you type a sentence into it, the model reads the sentence one piece at a time, consults its parameters, and writes the next most likely piece. It then writes the next piece after that, and the next, and the next, until it has written a complete answer.

The numbers inside the model come from a process called *training*, in which the model reads enormous quantities of text — books, articles, websites, code, transcripts — and slowly adjusts its numbers so that they predict the next word well. Training is expensive. It can cost millions of dollars and take weeks of computation on racks of specialized processors. Because training is so expensive, it cannot be done every day. So at some point, training stops, and the model is *frozen* — the numbers are saved to a file, and from that moment on, the model cannot learn anything new. It can still talk, but it can only talk using the knowledge it absorbed before the freeze.

Problem One: The Frozen Knowledge Cutoff

Suppose Lexi's training finished on January 31, 2024. If you ask her on March 15, 2025, "Who won the most recent Super Bowl?", she will answer based on what she saw before her freeze date. She will probably guess the Kansas City Chiefs, because they won the previous one. She will say it confidently. She will not warn you that her answer is more than a year old. This is the *frozen knowledge cutoff* problem: every LLM is, by construction, ignorant of the future past a certain date.

Children who grew up with smartphones sometimes find this hard to believe. "But you have the whole internet!" they say. No — the model had the whole internet once, and then it summarized the internet into weights. The weights are not the internet. They are a compressed, lossy, frozen snapshot of patterns the model noticed in the internet. Asking an LLM about events after its training cutoff is like asking a photograph of a library whether a particular book has been checked out. The photo does not know. It only knows what the library looked like when the shutter clicked.

RAG solves this in an elegant way: instead of trying to retrain the model, you simply give it a fresh page from today's news before it answers. The library is updated every morning. The model does not change. The model just reads a more recent page.

Problem Two: Hallucination

The second problem is more subtle. Sometimes Lexi is asked something she does not know, and instead of saying "I don't know," she makes up a confident-sounding answer. She might invent a Roman emperor named "Imperator Bellator III" who, according to her, ruled a small province in 312 BCE. There was no such emperor. Lexi just generated a sentence that sounded like a Roman emperor's name, with a date that sounded plausible, in a sentence structure that sounded authoritative. The pattern of "emperor name + reign date + province" was something she had seen many times during training, so she produced a new instance of that pattern. It sounded right. It was completely false.

AI researchers call this *hallucination*. The name is apt: the model is not deliberately lying, any more than a dream is deliberately lying. The model is filling a gap in its pattern-matching with the most statistically plausible continuation. It has no internal flag that says, "I do not actually know this; I am just guessing."

RAG dramatically reduces hallucination for questions whose answers exist in the library. If Lexi is asked about a Roman emperor who actually exists in our library, she reads that page and summarizes it. If she is asked about an emperor who does not exist in the library, the search returns no relevant pages, and the system can be designed to say, "I could not find this in the knowledge library," instead of inventing. So RAG does not eliminate hallucination entirely — clever users can still trick the system — but it provides a much firmer ground for the model to stand on.

Problem Three: Missing Private Information

The third problem is the one that matters most to most builders. Imagine your school has a 200-page handbook about its rules. Lexi has never seen this handbook. She cannot tell you what the rule is about wearing hats indoors, even though the answer is on page 47. Lexi also has never seen your medical records, your company's internal documentation, or the unpublished draft of your novel. These things live behind walls: logins, paywalls, file cabinets, and family group chats. They are not on the public internet, so they were never in Lexi's training data.

Fine-tuning — the alternative to RAG that we will compare in a moment — can teach a model new facts, but it is slow, expensive, and requires careful technical work. RAG, by contrast, simply requires you to *show* the model the document at the moment of asking. The model does not need to memorize anything; it just needs to be able to read.

This is why RAG has become the default way to connect language models to private knowledge. From law firms asking questions about millions of contracts, to hospitals looking up patient historie