What Is Context and Why Does It Matter?

What Is Context and Why Does It Matter?

The Memory Paradox

Imagine you're having a conversation with someone at a party. You introduce yourself, talk about your favorite video game for twenty minutes, discuss the plot of the latest movie you both watched, and then—right in the middle of your sentence—they look at you blankly and ask, "Sorry, who are you again?"

That's not just awkward. It's fundamentally broken communication.

Now imagine the reverse: you meet someone who has somehow read every book ever written, knows the plot of every movie, and can discuss the history of every civilization. But when you tell them your name, they forget it thirty seconds later. You could have the most knowledgeable person in the world standing in front of you, and the conversation would still be useless.

This is the memory paradox at the heart of modern AI—and understanding it is the key to understanding why some AI systems feel genuinely intelligent while others feel like expensive autocomplete machines.

Consider this: some AI models can read a 500-page novel and then answer detailed questions about the protagonist's motivations in chapter twelve. They can hold the entire text of *War and Peace* in their context and discuss Russian society with nuanced understanding. But other systems—sometimes the same model used differently—can't remember what they said thirty seconds ago in a conversation.

The difference isn't raw intelligence. The difference is *context engineering*.

This chapter will equip you to understand what context actually is in AI systems, why it creates the fundamental constraint that determines what's possible, and why engineering it properly is what separates a useful AI from a frustrating one. By the end, you'll have built the mental model that makes everything else in this book click into place.

What Context Actually Means

When you interact with an AI, you probably think you're just "asking it a question." But there's a lot more happening behind the scenes than a simple question-and-answer exchange. The AI system is making a prediction about what text should come next, and it makes this prediction based on something called the context.

Context is the complete set of information an AI system has access to while generating a response. This includes several distinct components, each playing a different role.

The Four Layers of Context

The conversation history is what most people think of when they imagine "AI memory." When you send a message and the AI responds, that exchange becomes part of the ongoing context. In a multi-turn conversation, the AI can reference earlier messages, recall what you were discussing, and build on previous responses. This is what makes chat interfaces feel conversational rather than like talking to a filing cabinet.

Uploaded documents and files form another crucial layer. Modern AI systems can accept images, PDFs, code repositories, spreadsheets, and other files as part of the input. When you upload a research paper and ask the AI to summarize it, that paper becomes part of the context—it's the information the AI is actually working with.

System instructions and prompts are the rules that govern how the AI should behave. These might specify the tone of responses ("Explain this as if you're teaching a teenager"), the format ("Always use bullet points for lists"), or constraints ("Never make up statistics"). These instructions are part of the context that the AI processes, even though you might not see them explicitly.

Prior knowledge from training is the most invisible layer. During training, the AI learned patterns from billions of examples—text from books, websites, conversations, and code. This knowledge is "baked in" to the model's parameters. When you ask GPT-4 about the French Revolution, it draws on this training knowledge, not on anything you explicitly provided.

Here's where it gets subtle: the AI doesn't actually "know" where its knowledge comes from. It doesn't distinguish between "I know this from the files you uploaded" and "I know this from my training." From the AI's perspective, all information in the context window is just... context. The distinction only matters for *you*, the engineer, because you need to understand what information the model can reliably access versus what it might hallucinate.

The Critical Distinction: Training Knowledge vs. Runtime Context

This distinction is so important that it's worth spending a moment really understanding it.

Training knowledge is what the model learned during its initial training phase. This knowledge has a timestamp—it reflects the state of the world when training ended. GPT-4's training cut-off is in early 2023, which means it doesn't "know" about events after that date unless you provide the information in the context. If you ask it about a movie released last month, it will either say it doesn't know or (worse) confidently make something up based on patterns from similar movies.

Runtime context is the information you provide *in the moment*. This could be a document you're analyzing, a conversation history from the past hour, or real-time data you just fetched from the internet. This information has no timestamp problem—it reflects the current state of whatever you shared.

This distinction creates a fundamental challenge: the model is equally confident (or equally uncertain) about information from training and information from context. It doesn't have a built-in sense of "I'm sure this is true because I read it in a reliable source you just uploaded" versus "I'm somewhat confident this is true because I learned it during training but I'm not 100% certain."

This is one of the core problems that context engineering addresses. When you engineer context well, you make it crystal clear to the model what information it should rely on. When you engineer it poorly, you create situations where the model confuses training knowledge with context information, or ignores relevant context entirely.

Think of it like this: if you give someone a book to read and then ask them questions about it, you expect their answers to come from the book. But what if they keep drifting back to things they learned in school years ago, ignoring the book entirely? What if they mix facts from the book with facts they already knew, without telling you which came from where?

That's the challenge with AI context. The model will happily "hallucinate"—generate confident-sounding but incorrect information—using a mix of training knowledge and context, without clearly signaling which source it's using. Good context engineering creates structures and prompts that minimize this confusion.

The Context Window: The Fundamental Constraint

Every AI system has a limit on how much information it can process at once. This limit is called the context window, and understanding it is essential to understanding why context engineering exists as a discipline.

What a Context Window Actually Measures

A context window is measured in tokens, which are roughly chunks of text. A token might be a complete word ("context" is one token) or part of a word ("ing" in "contexting" might be a separate token, or "tokenization" might be three tokens). On average, one token equals about four characters or three-quarters of an English word. So a 1000-token context can hold roughly 750 words—a few pages of text.

Different AI models have different context window sizes:

GPT-3.5 supports 4,096 tokens in its base version (about 3,000 words). This means the entire conversation—including all your previous messages and all the AI's responses—must fit within that limit. In a long conversation, you might run out of space after just a dozen exchanges.

GPT-4 typically offers 8,192 tokens in its base version and up to 128,000 tokens in the extended version. The 128K version can handle roughly 300 pages of text at once—about three novels.

**Cl