Why AI Makes Things Up — And How RAG Fixes It

Why AI Makes Things Up — And How RAG Fixes It

Imagine you're cramming for your history exam on the American Civil War. You open a chat window and type: "What were the main causes of the Battle of Gettysburg?" The AI answers you in flawless, confident paragraphs. It mentions General Lee, the Confederate advance into Pennsylvania, the strategic importance of the railroad junction, even the famous charge up Cemetery Ridge. The grammar is perfect. The structure is professional. It reads like it was pulled straight from a textbook.

There's only one problem: about a third of what it just told you is invented.

The names are mostly right. The general era is right. But specific details — the date of a skirmish, the number of casualties, a quote attributed to Lincoln that he never said — are stitched together from patterns the model has seen across millions of pages of text. The AI doesn't *know* Gettysburg the way your history teacher knows it. It knows what *sounds like* a Gettysburg explanation. And it has no way to tell the difference.

This is not a bug. It's how large language models work. And understanding *why* they work this way is the most important thing you can learn before you start building with them — because the entire field of RAG engineering exists to solve exactly this problem.

In this opening chapter, we're going to take that confident, polished, half-wrong answer apart piece by piece. You'll see exactly where the fabrications come from. You'll learn the difference between the kind of knowledge an AI *stores inside itself* and the kind it has to *look up*. And you'll build, with your own hands, the simplest possible version of the fix: a system that lets an AI check its work before it speaks.

By the end of this chapter, you'll have a working Python environment, you'll have watched a real model hallucinate in real time, and you'll have the mental model that grounds every chapter that follows. Let's begin.

---

1.1 The Polite Liar on Your Screen

Let's get more precise about what happened in that Gettysburg scenario. When you typed your question, the AI didn't go look anything up. It didn't open a database. It didn't cross-reference Wikipedia. It didn't even think, in any human sense of the word. What it did was far stranger, and far more mechanical.

It played a game of autocomplete — an extraordinarily sophisticated, billion-parameter autocomplete, but autocomplete nonetheless.

Here's the rule the model follows, in its simplest form: given all the text so far, predict what word (or, more precisely, what *token*) is most likely to come next. Then append that token to the text. Then repeat. The model doesn't have a "truthiness" detector. It doesn't have a fact-checking subroutine. It has one job: pick the next plausible token.

This single rule, scaled up to billions of parameters trained on hundreds of billions of words of human text, produces something that *feels* like understanding. It produces fluency. It produces grammar. It produces the cadence of a confident expert. But it does not, by itself, produce *truth*. Truth is not what the model is optimizing for. Plausibility is.

When a model generates a confident, fluent, completely made-up answer, we call that a hallucination. It's a deliberately soft word — the industry adopted it because "the AI confidently lied to you" sounded too accusatory for what is, technically, a side effect of how these systems work. But the effect on you, the user, is the same as if a human had lied. You end up believing things that aren't true.

Hallucinations come in a few flavors, and it's worth learning their names because you'll encounter them in every AI paper and product spec from now on:

• **Intrinsic hallucinations**: the model contradicts itself or the source it's drawing on. It says "the war began in 1861" and then two sentences later says "after the 1855 armistice." The information is internally inconsistent.
• **Extrinsic hallucinations**: the model says things that cannot be verified against any source — fabricated quotes, invented statistics, fake citations. These are the Gettysburg problem in pure form.
• **Factual hallucinations**: the model states something that's flatly wrong, often because it confuses two similar things. Ask it about a movie and it might mix up the plot of the sequel with the plot of the original. Ask it about a scientist and it might attribute Einstein's theories to Bohr.

Why does this happen? Three deep reasons, each of which we'll unpack:

First, the model is trained on a *snapshot* of the world. Training data is frozen at the moment the training run finishes. If the model finished training in March 2024, it knows nothing — *nothing* — about anything that happened in April 2024. It doesn't know about the new iPhone release, the latest election, or the scientific paper published last Tuesday. It can only pretend to know, and that pretense is exactly what produces hallucinations when you ask about recent events.

Second, the training data itself is messy. The internet contains truth, lies, satire, fiction, outdated facts, regional myths, and outright propaganda. The model learns from all of it without much ability to weigh which source is reliable. If a false fact appears on ten thousand web pages, the model will treat it as more plausible than a true fact that appears on a hundred pages — not because it's more true, but because it's *more statistically common*.

Third, and this is the one most people miss, the model has no internal fact-checker. When a human writes a sentence, part of our brain is constantly asking "is this true?" Large language models don't have that loop. They're not double-checking their work. They're not pausing to wonder. They're generating the next token, then the next, then the next, with the same mechanical rhythm from the first word to the last.

The result is what you saw at the start of this chapter: a polite, fluent, confident liar. And the bigger and more powerful the model, the *more* convincing the lies become, because the same machinery that produces fluency also produces the appearance of expertise. This is one of the central paradoxes of modern AI: capability and confabulation scale together.

---

1.2 What's Actually Inside the Model: Parametric Knowledge

To understand how RAG fixes the hallucination problem, you have to understand what kind of knowledge a language model actually possesses. Spoiler: it's a kind you've never encountered before, and it has properties that will surprise you.

When an AI model finishes training, all of its learned knowledge is stored in something called its parameters — billions of numerical values (weights) inside a neural network. Each weight is a tiny number, often something like 0.00347 or -0.12891. On their own, these numbers mean nothing to a human. But together, across billions of them, they encode everything the model has learned: how grammar works, how arguments are structured, what words tend to follow other words, and — crucially — vast amounts of factual knowledge about the world.

We call this parametric knowledge: knowledge that is stored *inside the parameters* of the model. It's baked into the weights during training. It's not a database you can query. It's not a file you can open. It's a giant soup of numbers that, when activated in the right way by the right input, produces the appearance of knowing things.

Here are the properties of parametric knowledge that matter most for what we're about to build:

It's frozen. Once training ends, the parameters don't change. They are, in a very real sense, a statue of what the model knew at one moment in time. If you want to update the model's knowledge, you have to either retrain the whole thing (expensive, slow, often impossible for huge models) or use techniques like fine-tuning (less expensive, but limited in scope). Either way, you can't just hand the model a new fact and expect it to know it ne