Foundations of Context in AI Systems

Foundations of Context in AI Systems

Imagine you are talking to a colleague who, despite being brilliant in many ways, suffers from a peculiar condition: at the start of every conversation, they forget everything they have ever learned, every project they have worked on, every preference you have expressed, and even the fact that you have met them before. The only thing they know in any given moment is the text you place in front of them right now. What they say next is entirely determined by that text—and nothing else.

That colleague is your large language model.

This is the single most important fact about every modern AI system you will ever build, and almost every problem you will encounter when building it traces back to a misunderstanding of this fact. A large language model (LLM) is not a thinking brain that you query like a database. It is a sophisticated next-token predictor whose behavior is entirely, exhaustively, and mercilessly shaped by the text it sees at the moment it generates its next response. The implications of this simple statement fill the rest of this book.

So what does this have to do with you, the engineer building LLM-powered applications? Everything. Because every interaction you design, every feature you ship, and every bug you debug will be a story about context—what text you gave the model, in what order, with what framing, and how much room was left for it to do its work. The discipline of designing that text—of deciding exactly what the model should see at inference time—is what this book calls context engineering.

In this opening chapter, we will build a foundation. We will reframe the LLM not as a brain but as a conditional generator. We will dissect the components that make up "context" in a typical AI system. We will introduce the constraint that governs everything: the finite token budget. We will see, through a simple chatbot example, how quickly context fills up and why naive approaches break down. And we will preview the five pillars of context engineering—long context, compression, summarization, retrieval, and state—that the rest of the book will explore in depth.

By the end of this chapter, you should be able to draw a complete context-flow diagram for any LLM application you encounter and point at exactly where the token budget is being spent. That diagram is the foundation on which every chapter that follows will rest.

The LLM Is a Conditional Next-Token Predictor

The phrase "next-token predictor" sounds technical, almost dismissive. People sometimes use it to imply that LLMs are simple statistical machines, just glorified autocomplete. That misses the point. A modern LLM is an extraordinarily capable next-token predictor—so capable that the ability to predict the next token well, given enough parameters and enough training data, produces something that looks like reasoning, knowledge, and even creativity. The point of calling it a next-token predictor is not to diminish it but to clarify exactly how it operates.

What a token actually is

At the lowest level, an LLM does not see characters, words, or sentences. It sees tokens—chunks of text, typically parts of words or common words, that have been mapped to integer IDs during training. When you write "The cat sat on the mat," the model might break this into tokens like ["The", " cat", " sat", " on", " the", " mat"]. Each token is converted to a number, and those numbers flow through the model's parameters. The output is a probability distribution over what token should come next.

Different models use different tokenization schemes, but the principles are similar. A token is not a word: a single word like "unbelievable" might break into two or three tokens (something like ["un", "believ", "able"]). A token is not a character: a token is typically a subword unit that balances vocabulary size against sequence length. Common words are usually a single token; rare words get split; numbers, punctuation, and whitespace follow their own rules. This is why counting tokens is not the same as counting words. The sentence "I love this!" might be 5 tokens in one model and 7 in another, depending on how the tokenizer handles exclamation marks, capitalization, and the contraction "I".

The tokenizer matters because it determines the cost, latency, and effective context length of every operation. We will dig into tokenization in detail in Chapter 2, but for now, the key insight is that tokens are the unit of accounting for everything in LLM engineering. Not words, not characters, not messages—tokens.

The conditional probability formulation

The critical word in our reframe, however, is not "next-token" but conditional. The model does not just predict the next token. It predicts the next token *given a context*. Formally, if we let $x$ represent the entire input text provided to the model, and we let $y$ be the next token to be generated, the model computes:

$$P(y \mid x) = \text{LLM}(x)$$

In English: the probability of each possible next token, conditioned on everything in the context $x$. The model's entire output is a sample from this distribution (or a deterministic selection like the most likely token, depending on the sampling settings). Every behavior the model exhibits—every answer, every refusal, every hallucination, every joke—is a function of this conditional probability distribution. Change the context, and you change the distribution. Change the distribution, and you change the behavior.

This is more than a mathematical formalism. It is the operational definition of how the model works. When the model produces a correct answer, it is because the context pushed the probability mass toward the correct tokens. When it produces an incorrect answer, it is because the context pushed the probability mass toward wrong tokens. There is no third option. The model is not lying, confabulating, or being stubborn; it is doing exactly what its conditioning allows.

Three immediate consequences for the engineer

This single formulation has three immediate consequences for the engineer, and they deserve to be stated plainly.

First, context is causation. When an LLM gives a wrong answer, it is never because the model "didn't know" in some abstract sense; it is because the context you provided did not contain the information needed to produce the right answer, or contained information that pushed the distribution toward a wrong one. The model is not lying, confabulating, or being stubborn. It is doing exactly what its conditioning allows. This is why debugging LLM applications is so often a context-debugging exercise: the bug is rarely in the model; it is in the input. Before you blame the model for a wrong answer, look at the context. Almost always, the answer is there.

Second, there is no memory outside the context window. Unlike a human who carries years of accumulated knowledge and recent experiences into a conversation, an LLM has no persistent state. Its "knowledge" is baked into its parameters during training, but those parameters never change at inference time. Everything it knows about *this* conversation, *this* user, *this* task, *this* document must be present in the context window at the moment it generates a response. Forget to include a fact in the context, and the model cannot use it. Include it but bury it under thousands of unrelated tokens, and the model will effectively forget it again—a phenomenon called "lost in the middle" that we will examine in detail in Chapter 4.

Third, the context is the only handle you have. You cannot reach into the model and adjust its "knowledge" or "personality" directly. The only way to influence the model's behavior is to change the context. This is why every LLM application is, fundamentally, an exercise in curating text. The system prompt, the conversation history, the retrieved documents, the tool outputs, the examples you include—all of these are tools for sha