The Sliding Window: Why Context Is the New Memory

Chapter 1: The Sliding Window — Why Context Is the New Memory

第 1 章：滑动的窗口 —— 为什么上下文即新记忆

1.1 The Detective and the Finite Notebook

1.1 侦探与那本有限的笔记本

The notebook was already half-full when Mira inherited it.

她继承这本笔记本时，它已经写满了一半。

She had been told there would be a notebook. What she had not been told — and what no one quite knew how to tell her — was how strange the notebook was. The city had decided, in a moment of optimism, to record everything: every conversation overheard on a bus, every invoice scanned at a corner shop, every text message sent after midnight by lamplight. The hope had been that if anyone ever needed to reconstruct the past, the past would be there, fully and faithfully, in writing. And so the city had built a scribe. The scribe listened, the scribe wrote, the scribe never slept.

Mira knew about scribes. She had worked with court stenographers, with surveillance analysts, with paralegals who transcribed depositions by the hour. She had never met a scribe quite like this one, though. This scribe did not transcribe after the fact — it listened in real time, and what it heard became the present moment of the city. The citizens did not so much live in the city as pass through the scribe, and the scribe made a record.

But the notebook had a number written in the bottom corner of its cover, the way certain ledgers do. It said 8192. Mira did not, at first, know what that number meant. She thought it was an inventory tag. It was not. It was the maximum number of pages the notebook could hold.

Three days into the case, Mira was reviewing a witness's account of a fight that had happened on a Tuesday evening in March. The witness was a bartender named Eli. He had given a long statement, and Mira had been cross-referencing it with the door-camera footage and the receipts from a nearby pharmacy — a man matching the suspect's description had bought adhesive bandages at 9:47 p.m. — and somewhere in the middle of the cross-reference, the notebook slid. There was no other word for it. The leftmost pages, the ones that contained the opening of Eli's statement, simply fell out from under her thumb and onto the floor.

"They're not lost," said the precinct archivist, a tired man named Charles who had been told to babysit the notebook. "They're not lost in any meaningful sense. They're just — not here anymore. The notebook has limits. We knew that. The pages that fall off are still the city's pages, the city's memories, the city's record. They just stop being in the notebook."

Mira stared at him. "If a page falls off the notebook, and I need that page to know who bought the bandages, what does it matter that the page exists somewhere in the city? I am holding the notebook. The notebook is all I have. The notebook is what the scribe is reading from when it tells me anything."

Charles said, "Then I suppose you had better hope the witness's most important part was not on those pages."

This is the book. This is the entire book, in a single image. Mira, a detective, is doing what every modern AI system does when it tries to think about the world: it is reading from a notebook that has a fixed number of pages, and the older pages keep sliding off. The notebook is the context window. The city whose history is being recorded is, in this metaphor, the universe of facts, events, and intentions the model is supposed to reason about. And the tragedy at the heart of every serious AI system built in 2024 and 2025 — and probably for several years beyond — is that the notebook is finite, the world is not, and the two facts are at war.

If you are reading this book, you have likely already felt the edges of the notebook. You have asked an LLM a question about a long document and watched it hallucinate the first paragraph. You have built an agent that performed beautifully for ten turns and then forgot the user's name. You have tried to summarize a meeting transcript and watched the summary, character by character, drift from the truth until it described a meeting that never happened. These are not edge cases. These are the boundaries of the context window announcing themselves.

The central question of this book is simple to state and brutally hard to answer: how do we engineer around a window that is finite when the world is not? More precisely: how should we engineer the memory, the compression, the summarization, the retrieval, and the state of an AI system so that it can think at human length, at human depth, and across human timescales — when its raw attentional bandwidth is bounded by a number like 8192, 32768, 200000, or even a million tokens, and the problems we want it to solve are essentially unbounded?

This chapter does not answer that question. It does something less ambitious and more important: it makes sure you can see the notebook, see it clearly, and see why everything else in the book exists because of it. We will follow Mira for a few more pages. We will then zoom out and look at the history that gave us this notebook — a history that, in many ways, chose the constraint. We will catalog the three ways the notebook betrays us: it loses pages, it corrupts pages, and it contaminates pages. We will survey the five engineering pillars that exist in 2025 for dealing with those betrayals. And we will end with a single picture — a four-layer Context Stack — that you can sketch on the back of a napkin and that will, if you carry it with you, organize the rest of this book for you.

Mira, for her part, will be back in the closing section. She will run out of pages at the worst possible moment. That moment is the bridge to Chapter 2, where we will ask the next question — not *whether* context is finite, but *what kinds of context* it should contain.

---

1.2 From Compressed Memory to Delivered Memory

1.2 从压缩记忆到投递记忆

To see why the notebook is finite, you have to know what the notebook replaced. Until roughly 2017, AI systems did not have a notebook at all — at least, not in the sense Mira is using. They had a different kind of memory, hidden inside themselves.

To understand this shift, we need to talk about two families of neural networks. The first is the recurrent neural network (RNN), and its more capable descendants, the LSTM and GRU. The second is the Transformer, the architecture underlying essentially every modern large language model. The difference between them is the difference between *remembering* and *being shown*.

In an RNN, the model processes a sequence one element at a time. It reads a word, updates an internal hidden state — a vector of numbers that summarizes everything it has seen so far — and then reads the next word. The hidden state is, in a meaningful sense, the model's memory. It is the only place the past exists. By the time the model finishes reading a thousand-word document, the document itself is gone; all that remains is a fixed-size vector, typically a few hundred to a few thousand floating-point numbers, that the network has *distilled* from the document. The information density of that vector is extraordinary. It is a compression of the document so severe that, in any literal sense, the document has been destroyed. Only its ghost remains, encoded in weights and activations.

This has a beautiful property and a terrible property. The beautiful property is that the model's memory cost is constant in the length of the input. You can feed an RNN a thousand words, a million words, a billion words, and the memory footprint of the model itself does not grow. The terrible property is that the distillation is lossy in a way that the network cannot easily audit. The hidden state is a bottleneck. Information that does not fit through the bottleneck is simply dropped, and the network has limited ability to look back at what it dropped, because what it dropped is no longer available anywhere. The RNN is, in this sense, a witness who was forced to testify from memory alone, and who gr