What Is Context and Why Should You Care?

What Is Context and Why Should You Care?

Picture this: you're in the middle of a lively conversation with a friend about planning a surprise birthday party. Your friend mentions "we should check that place near the old bookstore," and you immediately know exactly which place they mean—the cozy café with the blue awning where you both got caught in that rainstorm last spring. But here's the thing: if you were meeting someone new, someone who had no idea about your shared history, that same phrase would be completely meaningless. "Which place? What bookstore?" they'd ask, bewildered.

This everyday scenario captures the essence of what we call *context* in the world of artificial intelligence. Just as you carry your shared memories and experiences into every conversation with your friend, an AI model carries its *context* into every interaction. The difference is that you handle this naturally, almost unconsciously—your brain seamlessly pulls from years of shared experiences, inside jokes, and accumulated understanding. Teaching AI to do the same thing is what this book is all about.

Understanding What Context Actually Means

When we talk about context in AI systems, we're referring to all the information that an LLM (Large Language Model) has access to when generating a response. This includes the conversation history—what's been said before—the current question or request, any documents or data you've provided, and even subtle signals like formatting preferences or implied goals.

Think of context as the model's "workspace." When you send a message to an AI assistant, you're not just sending that single message in isolation. You're placing it into a workspace that may already contain previous messages, uploaded files, system instructions, and various other pieces of information. The model draws from this entire workspace to craft its response.

Consider a practical example that illustrates this beautifully. Imagine you're working with an AI to analyze a company's quarterly report. You might first upload a 50-page PDF of the report, then ask questions like "What was our revenue growth this quarter?" The model doesn't just look at your question—it looks at your question *plus* the entire report you uploaded. That's context in action.

Now, here's where it gets interesting: context isn't just about adding more information. It's about what the model *considers* when thinking through its response. When you ask a follow-up question like "And how does that compare to last year?", the model needs to remember your original question, the answer it gave, the report content, and now this new question—all simultaneously. Context is the bridge that connects all these pieces into a coherent understanding.

The Conversation as a Story

Every conversation with an AI is, in a sense, a story being told. Each message adds a new chapter. But unlike human memory, which is vast and fuzzy and associative, AI context has very specific boundaries.

The first message in a conversation sets the scene—it's like the opening paragraph of a story. Subsequent messages add to this scene, but there's a limit to how much "scene" can fit. This is fundamentally different from how humans remember conversations. If you ask me about a discussion we had three months ago, I might remember the key points even if I don't recall every exact word. An AI, without context from that conversation, has no such memory. To it, that conversation simply doesn't exist until you bring it back into the context.

This is why context management isn't just a technical detail—it's the difference between an AI that feels like a thoughtful collaborator and one that feels like a forgetful assistant who needs everything repeated.

What Context Contains

Let's break down the typical components of context into practical categories:

Conversation History forms the backbone of most AI interactions. This includes all previous messages in the current session, organized chronologically. When you say "continue from where we left off," the model uses this history to understand what "where we left off" actually means.

System Instructions are explicit directives you provide to shape the model's behavior. These might tell the model to explain things simply, to think step by step, or to focus on certain aspects of a task. A system instruction like "You are a helpful coding assistant" primes the model to respond in a specific way.

Attached Files and Documents can be uploaded directly into context. This might be a text file, a PDF, a spreadsheet, or code repository. When you upload a document, its contents become part of the context the model can reference.

Implicit Assumptions and Preferences are trickier. These aren't explicitly stated but can be inferred from the conversation. If someone has been asking about advanced mathematical concepts for an hour, the model might reasonably assume they're comfortable with complex explanations. But these inferences are limited and unreliable—which is why explicit context management often outperforms implicit assumptions.

Context Windows: The Invisible Boundaries

Now we arrive at one of the most crucial concepts in this entire book: the context window. If context is the information available to an AI model, the context window is the *maximum amount* of that information it can handle at once.

Think of it like this: imagine you're at a long meeting, and you have a notepad. The notepad has exactly 50 pages. You can write down everything important—key decisions, names mentioned, action items assigned—but once those 50 pages are full, something has to go. Maybe you stop writing down peripheral details. Maybe you start abbreviating. Or maybe you just hope you'll remember the important stuff without writing it down.

A context window works similarly. It's measured in *tokens*, which are roughly chunks of text (about 4 characters or ¾ of a word on average). Modern AI models have context windows ranging from roughly 4,000 tokens (about 3,000 words of text) to over 200,000 tokens (roughly 150,000 words—equivalent to a small novel) in the most capable systems available today.

Why Limits Exist

You might wonder: why can't we just make context windows infinite? The answer involves a mix of technical constraints and practical realities.

Computational Cost is the primary factor. Every token in the context window needs to be processed by the model when generating a response. The computational resources required grow roughly quadratically with context length—doubling the context window doesn't just double the processing time, it multiplies it by more than two. This makes very large context windows extremely expensive to operate.

Model Architecture also plays a role. While newer models can handle longer contexts than older ones, there's still a physical limit to how much information any neural network can effectively "attend to" when forming its response. Think of it like human attention: you can technically hear multiple conversations at a noisy party, but you can only really focus on one at a time.

Quality Degradation is perhaps the most practically important constraint. As context grows very large, models tend to pay less attention to information at the beginning and middle of the context, focusing more on recent information. This "lost in the middle" problem means that simply having more context doesn't guarantee better responses—effective context management becomes crucial.

A Concrete Example

Let's make this concrete. Suppose you're using an AI model with a 32,000-token context window. That might sound like a lot, but let's see how quickly it fills up:

Your conversation history: 20 messages averaging 200 tokens each = 4,000 tokens

A technical document you uploaded: approximately 8,000 tokens

System instructions and formatting requirements: 500 tokens

The current query and response: 1,500 tokens

That leaves you with about 18,