Why Your AI Keeps Making Stuff Up (And What a 'Knowledge Base' Has to Do With It)

Why Your AI Keeps Making Stuff Up (And What a 'Knowledge Base' Has to Do With It)

Your AI is lying to you. Not on purpose — but it is lying. Every chatbot you've ever used has, at some point, confidently told you something completely made up. The question isn't whether your AI will fabricate facts; it will. The question is what you're going to do about it.

The good news: by the time you finish this chapter, you'll understand exactly *why* this happens, why the obvious fixes ("just retrain it," "just paste in more documents," "just make the context window bigger") mostly don't work, and what the simple, elegant trick is that turns a lying machine into a useful one. That trick is called Retrieval-Augmented Generation, or RAG for short, and the rest of this book is about how to actually build one.

But before we get to the cure, let's look at the disease.

Three Times the AI Lied, and Once It Cost a Fortune

Let's start with three real disasters. They're funny in retrospect. They were less funny at the time.

The Lawyer Who Cased the Court

June 2023. A federal courtroom in New York. Steven Schwartz, a personal injury attorney, files what looks like a perfectly ordinary legal brief in a lawsuit against Avianca Airlines. The brief cites six previous court decisions to support his argument. There's just one problem: none of those cases exist. They have judge names, docket numbers, even quoted passages — all beautifully formatted, all utterly fabricated.

When opposing counsel points out that the cases can't be found in any legal database, Schwartz confesses. Yes, he used ChatGPT for legal research. Yes, he never verified the cases ChatGPT invented. And yes, when asked by the judge what he had to say for himself, he produced a sworn affidavit stating, in effect: "I was unaware of the possibility that ChatGPT could fabricate legal cases."

Judge P. Kevin Castel was not amused. Schwartz and his colleague Peter LoDuca were sanctioned $5,000 each. The judge noted drily that technological advances are not excuse for abandoning the duties of an officer of the court. Schwartz told the court he'd never used the chatbot before for legal filings and was, in his own words, "mortified."

What's the lesson here? Not that lawyers are dumb — Schwartz had decades of experience. The lesson is that ChatGPT did exactly what it was trained to do: it produced a plausible-looking block of text shaped like a legal argument. Plausible-looking text is what these systems are designed to emit. Truth is a side effect, when it happens at all.

The Ad That Erased $100 Billion

February 2023. Google is about to launch Bard, its AI chatbot competitor to ChatGPT. To drum up excitement, Google posts a short promotional GIF showing someone asking Bard: "What new discoveries from the James Webb Space Telescope can I tell my 9-year-old about?"

Bard cheerfully answers that the JWST "took the very first images of a planet outside of our own solar system."

There are several issues with this. First, the JWST didn't take the first pictures of an exoplanet. That distinction belongs to the European Southern Observatory's Very Large Telescope, which captured an image of a planet orbiting a brown dwarf way back in 2004. Second, this is a *promotional demo* by a company whose entire brand is "we organize the world's information." Third, this is being shown to journalists and investors as evidence that Google can compete.

The mistake was spotted quickly. Google's parent company, Alphabet, lost roughly $100 billion in market value the next day. An ad that had cost some money to make cost the company a hundred billion dollars in shareholder trust.

Why did Bard say it? Because in the vast soup of web pages and articles about the JWST, there are many sentences that look like the one Bard produced. "JWST took the first images of…" is a plausible-sounding sentence. Bard completed the pattern. The fact that it was wrong didn't matter to the model. The model doesn't have a fact-checking department. It has a sentence-completing department.

The Airline That Argued Its Own Chatbot Was Someone Else

February 2024. Jake Moffatt contacts Air Canada's customer service chatbot to ask about bereavement fares after his grandmother passed away. The chatbot helpfully tells him he can request a refund up to 90 days after purchase and apply for the bereavement rate retroactively. Moffatt, relying on this answer, books a full-price ticket and later asks for the bereavement discount.

Air Canada says no. They tell Moffatt the policy doesn't work that way. When Moffatt points out that their own chatbot told him otherwise, Air Canada's actual legal position — and I want you to appreciate the absurdity of this — is that the chatbot was "a separate legal entity" and they weren't responsible for what it said.

The Canadian Civil Resolution Tribunal was unimpressed. In a decision written like a very polite eye-roll, the tribunal noted that it would be "unreasonable" to expect customers to know they were interacting with a "separate entity," and ordered Air Canada to honor what its own chatbot promised. Air Canada was also told to pay Moffatt's tribunal fees.

This one is interesting because it shows the problem from the *company's* side. You can build the smartest chatbot in the world, but if it's making things up, your company is on the hook — not the chatbot.

What These Three Stories Have in Common

In every case, a real human with real authority — a lawyer, a Google marketing executive, an airline's customer service manager — trusted a thing the AI said. In every case, the AI didn't *know* what it was talking about. It just produced a fluent answer. The model isn't dishonest in any meaningful moral sense. It's just not built to care whether what it says is true. It's built to produce text that *looks* like the kind of text that would appear in response to the kind of question you asked.

And that's the entire problem in one sentence. Once you understand it, every hallucination, every "I don't know why it said that," every "$100 billion typo" makes sense.

Now let's pull back the curtain and figure out exactly what's happening inside the model when it lies.

Why Your AI Is Just a Very Confident Liar

Here is the deepest, simplest truth about how large language models work:

A large language model is a pattern machine, not a fact machine.

That's not a snide dismissal. It's the most precise one-sentence description of an LLM you can give. Let's unpack what it means, because once you see this, the rest of the chapter is going to click.

The Token Machine at the Heart of Everything

A large language model like GPT-4 or Claude or Llama doesn't read sentences the way you do. It doesn't parse grammar, build logical representations, look up facts in some internal encyclopedia. None of that happens.

What it does is something much simpler and, in its own way, much weirder. It takes a piece of text, breaks it into chunks called *tokens* (roughly: pieces of words), and predicts the most likely token to come next. Then it takes that token, appends it to the sequence, and predicts the *next* token. And the next. And the next. Until it has generated a response.

That's it. That's the whole engine.

To make those predictions, the model has been trained on a colossal amount of human-written text — much of the public internet, books, code, articles, Reddit comments, Wikipedia, the lot. Through that training, the model has built up billions and billions of numerical weights that, together, encode the statistical patterns of how words follow other words.

When you ask "What is the capital of France?", the model has seen "the capital of France is Paris" appear so many times in so many contexts that "Paris" becomes the overwhelmingly likely next token. It doesn't "know" Paris is the capital. It knows that "Paris" is the most statistically common word to appear after "the capital of France is."

When you ask "