The Architecture of LLM Comprehension

The Architecture of LLM Comprehension

When you craft a prompt and receive a response that misses your intent, the failure rarely originates from the model's knowledge or intelligence. More often, it emerges from a fundamental mismatch between how you framed your request and how the underlying architecture processes, interprets, and responds to text. Understanding why prompts succeed or fail requires peering beneath the surface of natural language into the mechanical realm where tokens flow through layers of mathematical operations, attention weights redistribute across context, and probability distributions collapse into discrete words. This chapter establishes that foundational understanding—the mental model that transforms prompt engineering from guesswork into principled design.

The central question driving our exploration is this: How do transformer architectures and attention mechanisms influence prompt effectiveness? To answer it, we must trace the path from input text through tokenization, through attention computation, through sampling decisions, and finally to output. At each stage, architectural choices create affordances and constraints that shape what the model can do with your prompt. Once you internalize these architectural realities, patterns that seemed arbitrary—why system prompts work, why few-shot examples improve performance, why placement within the context matters—reveal themselves as predictable consequences of how transformers process information.

The Token Prediction Foundation

Before any transformer architecture touches your prompt, the text must undergo tokenization—a transformation that breaks human-readable language into the discrete units the model actually processes. This seemingly mundane preprocessing step has profound implications for prompt design, yet it remains invisible to most practitioners working at the application layer.

Modern language models employ subword tokenization algorithms, most commonly Byte-Pair Encoding (BPE) or its variants. The fundamental insight of subword tokenization is that neither characters nor words alone make optimal processing units. Characters provide excessive granularity without capturing meaningful semantic units; words provide semantic coherence but create enormous vocabulary sizes and struggle with morphological variation, rare words, and cross-linguistic transfer. Subword tokenization strikes a balance by building a vocabulary from common character sequences observed in training data.

Consider what happens when you write a prompt containing the word "tokenization." Depending on the model's vocabulary, this might be encoded as a single token, as multiple tokens splitting at common boundaries, or even character-by-character if the word falls outside the vocabulary entirely. The token "token" and the suffix "ization" might each be individual tokens, or the model might split "token" into "tok" and "en" based on observed co-occurrence patterns. This fragmentation is deterministic for a given model but varies significantly across different models with different vocabulary sizes and training corpora.

The practical consequence of tokenization is that prompt length and semantic boundaries do not align with token count. A prompt that appears concise to human readers might consume a substantial portion of the context window due to how its words fragment into tokens. More critically, certain phrasings might accidentally place semantically critical words across token boundaries, weakening the model's ability to recognize the pattern you're trying to establish. For instance, if the phrase "not good" tokenizes differently than expected—perhaps "not" and "good" each receive heavy attention weights independently, rather than forming a cohesive negating unit—the model's attention mechanism might not process the negation as effectively as you intended.

The vocabulary size of a model directly impacts its tokenization behavior. Smaller vocabularies (around 30,000 tokens) fragment common words more frequently, while larger vocabularies (50,000+ tokens) can encode more complete words as single tokens. Some recent models employ byte-level tokenization with extremely large vocabularies, enabling them to represent virtually any text as a sequence of tokens without unknown-word handling. When designing prompts for specific models, awareness of these tokenization characteristics helps you anticipate how the model will parse your instructions.

Transformer Architecture and Self-Attention

With tokens defined, we can now examine how the transformer architecture processes a sequence to produce contextually informed representations. The transformer, introduced in the seminal "Attention Is All You Need" paper, revolutionized sequence processing by replacing recurrent operations with attention-based mechanisms that can process all tokens in parallel. Understanding this architecture is essential for principled prompt design.

The Self-Attention Mechanism

At the heart of the transformer lies self-attention, an operation that computes relationships between all pairs of positions in a sequence. For each token in the input, self-attention produces a new representation that incorporates information from other tokens, with the degree of influence determined by learned relevance patterns.

The computation proceeds through learned projection matrices that transform each token's embedding into three vectors: a Query vector representing what information this token is "looking for," a Key vector representing what information this token "offers," and a Value vector representing the actual information content to be retrieved. For a given token, the attention mechanism computes compatibility scores between its query and the keys of all tokens (including itself), applies a softmax transformation to normalize these scores into attention weights, and then computes a weighted sum of all tokens' value vectors using these weights.

Mathematically, for a sequence of tokens, the attention output for position $i$ is:

$$\text{Attention}(Q_i, K, V) = \sum_{j=1}^{n} \text{softmax}\left(\frac{Q_i K_j^T}{\sqrt{d_k}}\right) V_j$$

where $d_k$ is the dimensionality of the key vectors, used to prevent the dot products from growing too large during training. The softmax operation ensures that attention weights sum to 1 and creates a probability distribution over source positions.

The multi-head attention variant extends this by running multiple attention operations in parallel, each with different learned projection matrices. Each "head" can specialize in capturing different types of relationships—for instance, one head might attend to syntactic dependencies while another captures semantic similarity. The outputs from all heads are concatenated and projected to produce the final attention output. Modern large language models typically use dozens or hundreds of attention heads across many layers, creating an extraordinarily rich representational space.

Layer-by-Layer Processing

A transformer stack comprises multiple identical layers, each containing multi-head attention followed by feed-forward neural networks, with residual connections and layer normalization stabilizing training and propagation. The processing proceeds incrementally: the first layer computes attention patterns over the raw token embeddings, the second layer operates on the representations produced by the first layer, and so on through tens or hundreds of layers.

This layered processing creates a crucial phenomenon: different layers capture different levels of abstraction. Lower layers tend to encode surface-level patterns—syntactic relationships, local collocations, and token-level statistics. As representations propagate through layers, they accumulate broader contextual information. Upper layers encode abstract semantic relationships, high-level reasoning patterns, and task-relevant abstractions. For a language model trained on nex