Foundations of Context — What Machines Remember

Why does a large language model answer your question correctly one moment and confidently fabricate an answer the next? Why does a chatbot that performs flawlessly on a five-turn conversation begin to contradict itself when the dialogue reaches fifty turns? The answer, in nearly every case, lies not in some abstract measure of the model's "intelligence," but in a far more concrete, far more measurable quantity: the context it has been given. Context is the bounded set of information — tokens, embeddings, retrieved documents, system instructions, and prior turns of dialogue — that a model conditions upon when producing each new output. It is not merely an input. It is the operating substrate of intelligence, the workbench upon which all reasoning must physically take place. Change the context, and the model's behavior changes. Shrink the context below some critical threshold, and the model's apparent reasoning collapses into incoherence. This chapter opens the book by establishing a single, foundational claim: in modern AI, context is not a feature — it is the substrate of cognition, and engineering it is the central act of building intelligent systems.

The argument proceeds in six movements. We begin by giving context a precise definition, distinguishing it from the model itself and from the data on which the model was trained, and we visualize its structure as a bounded workspace. We then trace a brief historical lineage — from von Neumann's stored-program architecture through relational databases to the transformer — to show that the problem of "what does the machine have access to right now?" is as old as computing itself, and that each generation has answered it differently. We introduce the workbench analogy, a conceptual tool that will recur throughout the book, in which the context window is a finite physical surface and reasoning is the act of arranging objects upon it. We then formalize the distinction between two fundamentally different kinds of memory — parametric memory, stored in the model's weights, and episodic memory, supplied at inference time — and examine how their interaction produces both the capabilities and the failure modes we observe. We survey the consequences of context scarcity, including the now-famous *lost-in-the-middle* phenomenon, in which models systematically fail to attend to information buried in the center of long contexts. Finally, we walk through a worked case study of a customer-service chatbot, observing in concrete terms how a system that performs admirably at small scale begins to fail as its conversation history grows, and what interventions a skilled context engineer would apply. By the end of this chapter, you should be able to draw a labeled diagram of an AI system, point to the context, and articulate in formal language why its size is a binding constraint on reasoning quality.

1.1 Context as a Bounded Information Substrate

To speak precisely about context, we must first say what counts as context and, just as importantly, what does not. A common source of confusion — even among experienced practitioners — is the conflation of three distinct quantities: the model, the training data, and the context. Each of these plays a different role in producing an output, and conflating them leads to misdiagnosis of system failures.

The model, in this book, refers to the parameterized function — typically a deep neural network with hundreds of billions of floating-point weights — that maps an input sequence to an output sequence. The model is fixed at inference time. It does not change when you ask it a question. It is the "machinery" of the system, akin to the CPU and arithmetic logic unit of a classical computer. The model is the same model, second after second, query after query, session after session.

The training data refers to the corpus of text, code, and other modalities from which the model's parameters were learned during the pretraining process. By the time a model is deployed, its training data exists only indirectly — as statistical traces encoded in the weights. The model cannot "look up" a passage from its training data the way a database can look up a row. It can only produce outputs that are statistically consistent with patterns it absorbed during training. This is a subtle but crucial distinction: the training data is the *history* of the model, not its *current information*.

The context, finally, is the set of information that is supplied to the model at the moment of inference — that is, at the moment a specific output is being produced. It is the only information the model "sees" when generating its response. It is constructed freshly for every query, and it is the only thing that varies from one moment to the next in the operation of a deployed model.

Formally, we can write the model's behavior as a conditional distribution:

$$P(\text{output} \mid \text{context})$$

The model takes the context as input and produces a probability distribution over possible continuations. The context itself is typically a sequence of discrete tokens drawn from a finite vocabulary, and the model processes this entire sequence — every token, in order — to produce each new token of its output. We will return to the precise mechanics of this process in Chapter 2, when we examine tokenization. For now, the essential point is this: the model is a fixed function, the training data is its history, and the context is its present.

What, then, is included in the context? In a modern AI system, the context is almost always a heterogeneous mixture, assembled from several distinct sources that compete for the same finite resource:

1. System instructions. A prepended block of text that tells the model how to behave — its role, its tone, its constraints, its output format. For example: "You are a helpful assistant. Answer concisely. If you do not know the answer, say so. Never invent statistics." 2. Retrieved documents. External information fetched from a database, search engine, or vector store and inserted into the context to ground the model's response in factual material it might not have memorized during training. This is the foundation of the technique known as Retrieval-Augmented Generation (RAG). 3. Conversation history. The running record of the dialogue between the user and the model, including both the user's prior messages and the model's prior responses. 4. Tool outputs. Results returned from external function calls — the output of a calculator, the response from an API, the result of a database query — that the model has chosen to invoke during its reasoning process. 5. Few-shot examples. Carefully chosen demonstrations inserted into the context to illustrate the desired behavior, a technique known as *in-context learning*. By showing the model several examples of the task before asking it to perform a new instance, the engineer can shape behavior without changing parameters. 6. The current query. The user's most recent message, possibly reformulated, expanded, or augmented with clarifications, which the model is asked to respond to.

All of these sources compete for the same finite resource: the context window, the maximum number of tokens the model can process in a single forward pass. The context window is the model's perceptual horizon. Anything outside it is, for all practical purposes, invisible to the model. This boundedness is the central physical constraint on the system, and it is the reason context engineering exists as a discipline.

To make this concrete, imagine a model with a context window of 8,192 tokens — a typical size for production systems in the early 2020s. Suppose the system prompt alone consumes 500 tokens. Suppose three retrieved documents, each averaging 1,500 tokens, are inserted: 4,500 tokens. Suppose the conversation history has grown to 2,000 tokens. The current query adds 200 tokens. The total is 7,2