The Lying Oracle: Why a Brilliant AI Keeps Getting Things Wrong

The Lying Oracle: Why a Brilliant AI Keeps Things Wrong

The freshman was crying.

Maya Chen had prepared for many possible disasters on launch day. She had prepared for the Wi-Fi to fail. She had prepared for the school's firewall to block the API endpoint. She had even prepared, in a small dark corner of her mind, for her club members to lose interest and for the project to fizzle in a cloud of unmet deadlines.

She had not prepared for Emily Patterson to walk into the computer lab at 3:47 p.m. on a Thursday with tears in her eyes and a printout in her hand.

"It's wrong," Emily said, sliding the printout across the desk. "Everything is wrong."

Maya looked at the paper. It was a screenshot of the chatbot's chat window, printed in the school's frugal black-and-white. The chatbot was called AskMrHistory, and it was supposed to be the pride of Westbrook High's AI Club, a friendly assistant that any student could ask about American history. Maya had spent four months building it, mostly with help from her computer science teacher, Mr. Vance, and a small grant from the school's STEM budget.

The screenshot showed this exchange:

> Emily: When did the American Civil War end? > > AskMrHistory: The American Civil War concluded in 1987, following the signing of the Treaty of Appomattox Reinstated. The conflict, which had lasted nearly 125 years, finally ended when the remaining Confederate holdouts in the southern Appalachians surrendered to federal forces. You can read more about this in Dr. Jonathan Whitfield's seminal paper, *"The Long Reconstruction: Conflict Termination in 20th Century America"*, published in the *Journal of American Historical Studies*, Vol. 47, Issue 3, 1989.

Maya stared at the words.

1987. A one-hundred-and-twenty-five-year Civil War. A treaty that did not exist. A journal article that, she was fairly sure, had never been written by anyone named Jonathan Whitfield, because she had just searched the school's library catalog and found nothing, and a quick mental check told her that the *Journal of American Historical Studies* did not have a Volume 47 because the journal had only been founded in 1962 and was now in its early sixties.

The Civil War, of course, had ended in 1865. Maya knew this the way she knew that water was wet and that the cafeteria's pizza was a culinary crime. The war had ended at Appomattox Court House in Virginia, with Robert E. Lee surrendering to Ulysses S. Grant, and then there had been the long, painful period of Reconstruction, and then there had been Jim Crow, and then there had been the civil rights movement, and the rest was a complicated story that historians were still arguing about.

But 1987? A treaty? A paper?

"I told my mom I was going to use the chatbot for my homework," Emily said, her voice wobbling. "She saw the screen over my shoulder and said that couldn't be right. And then I tried to find the paper and it's not real. Is your chatbot broken?"

Maya wanted to crawl under the desk. She wanted the floor to open up and swallow her whole, her laptop, her club, her four months of work, all of it.

"It's not broken," she said carefully. "It's just... um. It's doing something else. I'll figure it out. I'm really sorry."

"I'm going to go use Wikipedia," Emily said, and walked out.

Maya sat alone in the lab for ten minutes, staring at the screenshot. Then she pulled out her phone and texted Mr. Vance.

`emergency. freshman cried. chatbot invented a civil war. please help`

The reply came in under a minute.

`On my way. Don't touch anything.`

The Closed-Book Brain

Mr. Vance arrived in eight minutes, which was fast for a man who taught on the third floor and parked in the back lot. He was carrying a cup of coffee and a copy of *The New Yorker* that he had clearly been planning to read during his planning period. He set both down, pulled up a chair next to Maya, and asked her to show him the failure.

"Read it aloud," he said. "Slowly."

Maya read the chatbot's response. She watched Mr. Vance's face as she did. He did not look shocked, exactly. He looked like a doctor who had been called to the house for the first time but had seen the symptoms many times before.

"1987," Mr. Vance repeated when she finished. "Treaty of Appomattox Reinstated. Dr. Jonathan Whitfield."

"None of it's real," Maya said. "I checked. The treaty doesn't exist. The journal article doesn't exist. Whitfield doesn't exist."

"No," Mr. Vance agreed. "None of it is real. And yet, look at the style. Look at how confident it sounds. Look at how it even invents a citation in exactly the format that a real citation would have."

He was right. The chatbot's response had the cadence of a real answer. It had a journal name. It had a volume number. It had an issue number. It had a publication year. It even had a title for the paper, in quotation marks, italicized correctly. From a distance, it looked like scholarship. Up close, it was a very expensive, very confident hallucination.

"Maya," Mr. Vance said. "I need to ask you a question, and I need you to think about it carefully before you answer. What does it actually mean when we say a model 'knows' something?"

Maya opened her mouth, then closed it. She had been building AI systems for four months. She had watched dozens of YouTube tutorials. She had read a book about prompt engineering. And she realized, with the uncomfortable clarity that only comes in moments of public humiliation, that she did not have a good answer.

"I mean... it learned it from the internet?" she tried.

"It learned the *patterns* of language from the internet," Mr. Vance said. "It learned that the words 'Civil War' tend to be followed by words like '1861' and '1865' and 'Appomattox.' It learned that when you ask a question about history, you tend to get a response that sounds like an encyclopedia entry. It learned that academic citations have a particular shape. But it did not store the facts themselves in any way you would recognize as a database. It stored them as weights inside a giant mathematical function."

He paused to let that sink in.

"Think of it this way," he continued. "If you studied for the most important exam of your life by reading every book in the world exactly once, what would you remember a year later?"

Maya thought about it. "Some of it," she said. "I'd remember the parts that surprised me, and the parts that came up a lot. But not everything. And I might remember things wrong."

"Exactly. And a large language model is even worse at this than you are, because you at least have a memory system that was designed by evolution to store specific episodes. The model doesn't. It has a giant block of numbers, and when you ask it a question, it does its best to *generate* an answer that fits the patterns it has seen. Sometimes that answer is true. Sometimes it isn't. And when it isn't, it has no way to know, because it doesn't have a separate memory it can check against."

Maya sat with this for a moment. "So when it said 1987," she said slowly, "it wasn't *lying*. It was... guessing?"

"It was doing something more interesting than guessing," Mr. Vance said. "It was *completing a pattern*. The question 'when did the Civil War end' activates a huge cloud of related patterns inside the model: dates, names, places, books, papers, treaties. The model picks the next most likely word, then the next, then the next. Most of the time, those words line up with reality. But sometimes, when the question is unusual, or when the model is missing some piece of the puzzle, the pattern completion goes off the rails. And what comes out is grammatically perfect and factually wrong. We call that a *hallucination*."

He tapped the screenshot.

"And the worst part is, the model has no internal flag that says 'I don't know.' There is no little light inside its head that turns on when it ventures into fiction. It generates the words '1987' with the same confidence it would use to generate the words '