The Whisper Problem / 低语者的困境

The Whisper Problem / 低语者的困境

1.1 Lin's Bot: A Modern Fable

It was 11:47 PM on a Thursday when Lin typed her last Slack message of the night. She had been at her desk in Hangzhou's Future Sci-Tech City for fourteen hours. The cursor blinked in front of her, accusing her. Her prototype — a customer service agent named 晓语 ("Little Whisper") — had just failed its third demo of the week, and the chief product officer had finally used the words nobody wants to hear in a Series A startup: *this isn't working*.

The product was simple. Too simple, perhaps, which is why it was supposed to be easy. 小语 would be the front door of an insurance claims platform. A user would explain their problem, the bot would ask follow-up questions, gather the relevant information, and route the case to a human adjuster with a tidy summary attached. The market for such bots was enormous. The competition was fierce. The deadline was six weeks away.

Lin was not a novice. She had shipped three LLM-powered features in the previous year — a writing assistant for a publishing house, a code-review bot for an internal toolchain, and a RAG system for a legal-tech company. She knew how to call an API. She knew how to write a system prompt. She knew, in the abstract, that context windows were finite. What she did not yet know — what no amount of API documentation had prepared her for — was how *aggressively* a transformer would forget.

The first version of 小语 used a sliding window. The last ten messages of the conversation were concatenated into the prompt. The system prompt was roughly 400 tokens. Each user turn was 50–200 tokens. The model's context window was 8,000 tokens, plenty of room. Lin's early tests were promising. A user would say *"I was in a car accident last Tuesday on the G92 highway"* and the bot would respond intelligently, asking which exit, whether there were injuries, whether police were involved. The summaries generated for human adjusters were crisp and accurate.

Then came the third turn. And the fourth. And the eleventh.

A user named Mr. Wang, testing the system on a Saturday morning, typed:

> Turn 1: My mother was in a car accident. She is 71 years old. She has a heart condition and takes warfarin. *(用户：我的母亲出了车祸。她71岁，有心脏病，长期服用华法林。)*

> Turn 2: It happened yesterday afternoon in Suzhou, on the second ring road near the Yangtze bridge. *(用户：昨天下午在苏州，靠近长江大桥的二环路上。)*

> Turn 3: I have the police report number. It is SU-2024-008813. *(用户：我有事故责任认定书编号，SU-2024-008813。)*

By turn 6, the user had explained that the other driver ran a red light, that his mother was transported to the First Affiliated Hospital of Suzhou University, and that she was currently in stable condition with a fractured wrist and bruised ribs.

> Turn 7: So given all of this, can the bot summarize the case for the adjuster?

Lin watched the screen. The summary that came back was fluent, polite, well-structured — and almost entirely wrong.

> *"The user is reporting a vehicle incident. The user's mother is elderly. The user mentions Suzhou as a possible location. The user is requesting a summary for an insurance adjuster. The user has not yet provided a police report number or details about injuries."*

Everything Mr. Wang had said in turns 1 through 6 — the age, the medical history, the warfarin, the report number, the hospital, the injuries — was gone. The bot had not lost it the way a human loses track of a thread. It had lost it the way a whiteboard is erased. There was no degradation, no fuzziness, no graceful approximation. There was simply *nothing there*.

Lin stared at the screen. She scrolled up. The information was in the history. She scrolled down. The information was not in the response. She opened the API call inspector, looked at the messages array. All seven turns were there. She looked at the system prompt. The summary instructions were clear. She ran the same call again. Same wrong answer.

She did what any of us would do. She blamed herself. She rewrote the system prompt to be more emphatic: *"You MUST include all factual details from the conversation in your summary. Pay particular attention to medical history, report numbers, and locations."* The next test produced a summary that included *more* of the information but missed the warfarin. She added *"especially any mention of blood thinners or anticoagulants."* This time it got warfarin but lost the hospital. She added the hospital. It lost the report number.

She was, without knowing the term for it yet, playing a game of *attention Whac-A-Mole*. Every time she emphasized one fact, the model dropped another. The total information density of her context was not increasing. It was *bouncing*, like beads on a tilted tray. She could make the beads cluster in different places, but she could not make more of them fit.

That night, after the failed demo, Lin walked home along the canal. The October air was cool. The bridges over the water were lit in soft blue. She thought about the problem. She thought about the system prompt. She thought about a metaphor her undergraduate advisor had once used, in a different context entirely, about LLM behavior:

> *"A transformer does not read. It listens. And like anyone listening to a long speech, it remembers best what was said first, best what was said last, and worst what was said in the middle. If you want it to remember the middle, you have to make the middle into a beginning or an end."*

Lin did not yet know that this observation, stripped of its metaphor, was about to become the central problem of her professional life — and of a discipline that, in 2024, was just beginning to have a name: Context Engineering.

This book begins with Lin's bot because every builder of LLM applications eventually meets her bot. The customer service version. The therapy companion version. The coding assistant version. The research agent version. They are all, in some sense, the same bot, and they all eventually whisper their most important information into a window that is too small, that has too much else in it, and that forgets with perfect, indifferent efficiency.

The remainder of this chapter explains *why* this happens. We will look at the paradox of a model that contains the entire internet in its weights but remembers nothing between turns. We will walk through three historical epochs of chatbot memory — ELIZA, the RNN era, and the modern transformer era — and see that the forgetting problem is not a bug but a consequence of how these systems fundamentally work. We will examine the empirical evidence that simply enlarging the context window is not the answer, because attention has a budget and that budget is brutally non-uniform. We will introduce the discipline that emerges from confronting these constraints. And we will end with a small case study and a small exercise, the way every chapter in this book will end: with something concrete enough that you can do it on a Sunday afternoon.

If you have ever built an LLM application that mysteriously lost track of a fact mid-conversation, you have already met the whisper problem. Let us make it precise.

---

1.2 The Paradox of the Empty Brain

The central paradox of large language models, stated as plainly as possible, is this:

> An LLM contains more readable knowledge than any single human has ever possessed, and remembers less than a goldfish.

A modern frontier model has been trained on somewhere between ten and fifteen trillion tokens of text. If you printed that text out at 250 words per page, the stack would reach higher than the orbit of the International Space Station. The training corpus contains, in some form, the entire English Wikipedia, large fractions of GitHub, PubMed, arXiv, court records, product manuals, religious texts, romance novels, and a non-trivial amount of internet argument. A sufficiently capable model can be queried, in natural language, about any fact contained in that corpus, and will pro