The Architecture of Understanding: How LLMs Process Prompts

In the art of prompt engineering, understanding *how* a Large Language Model (LLM) processes your input is not an academic exercise; it is the fundamental prerequisite for moving from trial-and-error to deliberate, predictable construction. Every token you type, every space you insert, and every structural choice you make is translated into a cascade of mathematical operations within a high-dimensional space. This chapter establishes the rigorous mental model required to bridge the gap between the human intent behind a prompt and the model's computational interpretation. We will deconstruct the journey of a prompt—from a string of characters to a generated sequence of tokens—by examining the core architectural components of transformer-based models: tokenization, embedding spaces, the attention mechanism, and autoregressive decoding. The focus is on linking prompt structure to model computation to explain *why* certain patterns succeed or fail. We will explore concrete phenomena like positional bias, the quantifiable impact of prompt length on attention distribution, and the architectural roots of "prompt injection." Finally, we will contextualize this mechanics within the broader ecosystem of model scale, training data cutoffs, and fine-tuning methodologies (instruction tuning, RLHF), which fundamentally shape a model's responsiveness to prompts. This chapter is your map of the terrain; subsequent chapters will teach you how to navigate it with precision.

1.1 From Text to Numbers: The Tokenization Pipeline

Before any semantic processing can occur, the raw string of your prompt must undergo a deterministic transformation into a sequence of integers, known as token IDs. This is the domain of the tokenizer, a critical and often underestimated component that acts as the model's sensory organ for language. The tokenizer is not a generic component; it is a fixed, pre-trained artifact bundled with the model (e.g., `GPT-2`'s `BytePairEncoding`, `Llama`'s `SentencePiece`). Its vocabulary size—typically ranging from 32,000 to over 250,000 tokens—defines the model's basic "alphabet." A token can represent a whole word ("hello"), a subword fragment ("ing"), a single character, or even a byte. This subword approach (BPE, WordPiece, Unigram) is a practical compromise between the combinatorial explosion of a word-level vocabulary and the inefficiency of a character-level one.

The Practical Impact on Prompt Design: Tokenization has immediate, tangible consequences for prompt engineering. 1. Token Efficiency: Different phrasings of the same semantic concept can have radically different token counts. For example, the word "unbelievable" might be a single token in one model but split into `["un", "believ", "able"]` in another. Using longer, more complex words can consume more of the model's finite context window (e.g., 4k, 8k, 128k tokens), leaving less room for the actual task context and examples. Actionable Insight: When operating near a model's context limit, prefer shorter, more common words that are likely to be single tokens. You can use the model's tokenizer (e.g., via Hugging Face's `transformers` library) to audit your prompt's token count and composition. ```python from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf") prompt = "The phenomenon of prompt injection reveals a fundamental aspect of transformer architecture." tokens = tokenizer.encode(prompt) print(f"Token count: {len(tokens)}") print(f"Token IDs: {tokens}") print(f"Decoded tokens: {tokenizer.convert_ids_to_tokens(tokens)}") ``` This code reveals the exact granularity the model works with. You might discover that "phenomenon" splits into `["phen", "omen", "on"]`, wasting three token slots.

2. Whitespace and Punctuation: Tokenizers treat whitespace and punctuation explicitly. A leading space often creates a separate token. In many tokenizers, common punctuation like periods or commas are standalone tokens. This sensitivity means that `"Hello world"` and `"Hello world."` have different token sequences. Case Study: Early GPT models were notorious for being "space-sensitive." A prompt like `"Translate to French: Hello"` might be tokenized differently than `"Translate to French:Hello"` (missing space after colon), potentially leading to inconsistent performance. Modern chat-optimized models are more robust, but the principle remains.

3. Out-of-Vocabulary (OOV) and Byte-Level Fallbacks: Modern tokenizers, especially those designed for multilingual or code-heavy use (like `GPT-4`'s `tiktoken`), often use a byte-level fallback. If a character sequence is not in the vocabulary, it is decomposed into its constituent bytes (UTF-8). This prevents hard OOV errors but can lead to longer, less meaningful representations. A rare Unicode symbol might be represented by 4-6 byte tokens, severely impacting context efficiency.

The tokenization stage is a lossy, irreversible first filter. The model never sees your original string; it only sees the integer IDs that emerge from this process. Therefore, the first law of prompt engineering is: You are not writing for humans; you are writing for a tokenizer. Your mental model must account for this discrete, statistical lens.

1.2 Embedding Spaces: Mapping Tokens to Meaning

Once tokenized, each token ID is mapped to a high-dimensional vector, typically of dimension $d_{model}$ (e.g., 768 for `BERT-base`, 4096 for `GPT-3.5`, 12,288 for `GPT-4`). This vector, called a token embedding, is a point in a dense, continuous space where geometric relationships approximate semantic relationships. This embedding matrix $W_e \in \mathbb{R}^{V \times d_{model}}$ (where $V$ is vocabulary size) is a core learned parameter of the model.

However, a raw embedding lacks two critical pieces of information: 1) The position of the token in the sequence, and 2) The fact that it is part of a larger sequence. Transformers are inherently permutation-invariant; without positional information, "dog bites man" and "man bites dog" would have identical internal representations. To counteract this, positional encodings (sinusoidal in the original Transformer, learned in models like BERT) or positional embeddings (learned vectors added to token embeddings) are injected. The resulting vector for the $i$-th token is: $$h_i^{(0)} = \text{Embedding}(token_i) + \text{PositionalEncoding}(i)$$ This $h_i^{(0)}$ is the initial input to the first layer of the transformer stack.

The Geometry of Prompting: The embedding space is where the "magic" of analogy and meaning resides. Classic experiments like `king - man + woman ≈ queen` are operations in this space. For prompt engineers, this implies:

• **Semantic Proximity:** Using synonyms or paraphrases will place tokens in nearby regions. However, due to the high dimensionality and complex training objectives, "nearby" is not always intuitive. The model's notion of similarity is derived from co-occurrence statistics in its training corpus.
• **Conceptual Anchoring:** The first few tokens of your prompt set a strong initial context. Their embeddings establish a region of the embedding space that subsequent tokens will be processed relative to. This is the architectural basis for **system prompts** (e.g., `"You are a helpful assistant."`)—they prime the entire representation space for the interaction.
• **Embedding Superposition:** Research has shown that individual neurons or subspaces in these embeddings can encode multiple, sometimes conflicting, concepts. A token's representation is a weighted sum of many latent features. This is why a model can simultaneously understand "bank" in the contexts of "river bank" and "financial bank" within the same prompt—different attention heads can isolate different features of the shared embedding.

1.3 The Attention Mechanism: The Core Computational Engine

Th