Foundations of Large Language Models

The journey into prompt engineering begins not with the prompt itself, but with a deep understanding of the systems that process them. Large Language Models represent one of the most significant technological advances in artificial intelligence, yet their effectiveness as tools for human interaction depends fundamentally on how we communicate with them. To master prompt engineering is to understand not merely what to say to these models, but why certain formulations succeed while others fail—and this understanding requires examining the underlying architecture, training methodology, and emergent behaviors that define modern language models.

The transformer architecture, introduced by Vaswani and colleagues in their seminal 2017 paper "Attention Is All You Need," fundamentally reshaped the landscape of natural language processing and eventually became the foundation for virtually all modern large language models. Understanding this architecture in its evolving forms provides essential insight into the capabilities and limitations that prompt engineers must navigate. The original transformer design employed a encoder-decoder structure, where the encoder processes input sequences into continuous representations and the decoder generates output autoregressively, one token at a time. This architectural choice, while revolutionary, has undergone substantial refinement as researchers discovered that pure decoder-only architectures—with modifications—often prove more effective for language generation tasks.

At the heart of the transformer lies the self-attention mechanism, a computational pattern that allows each position in a sequence to attend to all other positions, computing a weighted representation based on learned relationships. The mathematical formulation begins with the computation of query, key, and value projections for each token:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

where $d_k$ represents the dimensionality of the key vectors, and the scaling factor $\sqrt{d_k}$ prevents the dot products from growing too large in high dimensions, which would push the softmax function into regions with extremely small gradients. This mechanism enables the model to capture long-range dependencies that traditional recurrent architectures struggled to maintain, fundamentally addressing the vanishing gradient problem that plagued earlier sequential models.

The multi-head attention variant extends this basic mechanism by running multiple attention operations in parallel, each with learned linear projections to different subspaces of the representation space:

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)W^O$$

where each $\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)$. This parallel attention allows the model to attend to different aspects of the relationships between tokens simultaneously, with each head potentially learning to capture distinct linguistic phenomena—some heads might focus on syntactic dependencies, others on semantic relationships, and still others on coreference or positional patterns.

However, the computational complexity of full attention scales quadratically with sequence length, growing as $O(n^2 \cdot d)$ where $n$ is the sequence length and $d$ is the model dimension. This quadratic scaling presents substantial challenges for processing long documents, limiting practical context windows and driving the development of more efficient attention variants. The evolution from vanilla attention to modern sparse and efficient attention patterns represents one of the most consequential developments in transformer architecture optimization.

Rotary Position Embeddings (RoPE), introduced by Su et al. in their 2021 paper, represent a fundamental shift in how positional information is incorporated into transformer models. Unlike absolute position encodings added to token embeddings, or relative position embeddings that modify the attention computation directly, RoPE encodes position by rotating the query and key vectors in the embedding space. The rotation is applied in 2D subspaces corresponding to different dimensions, with rotation angles proportional to the position:

$$\mathbf{q}_m^{(i)} = \mathbf{R}_{m,d_i}\mathbf{q}_i, \quad \mathbf{k}_n^{(i)} = \mathbf{R}_{n,d_i}\mathbf{k}_i$$

where $\mathbf{R}_{m,d_i}$ represents a rotation matrix in the $i$-th subspace spanning dimensions $d_i$. This elegant formulation provides inherent decay of attention scores with increasing relative distance—a property that matches the empirical observation that distant tokens should generally contribute less to each other's representations. RoPE has become the dominant position encoding scheme in open-source models, with implementations in LLaMA, Mistral, and numerous other architectures, because it enables efficient extrapolation to sequence lengths not seen during training.

Grouped-Query Attention (GQA) emerged from research at Google as a middle ground between standard multi-head attention and multi-query attention, which uses a single key-value pair for all attention heads. In standard MHA, each head maintains its own query, key, and value projections. MQA reduces inference costs by sharing keys and values across heads, but at the cost of model quality. GQA interpolates between these extremes by grouping multiple query heads to share single key-value heads. If we have $n_q$ query heads and $n_{kv}$ key-value heads where $n_{kv} < n_q$, each key-value head is shared by $n_q / n_{kv}$ query heads. This approach preserves much of the representational richness of MHA while substantially reducing the memory bandwidth required during inference, since the key-value cache during autoregressive generation scales with $n_{kv}$ rather than $n_q$.

Sparse attention patterns extend this efficiency drive further, attempting to approximate full attention with selective connectivity. The Sparse Transformer introduced by Child et al. demonstrated that strided attention patterns—where each token attends to tokens at fixed intervals—could match full attention performance on many tasks while dramatically reducing computational requirements. More sophisticated patterns include the sliding window attention used in Mistral and other models, where each token attends only to a fixed window of preceding tokens plus itself. This local attention captures immediate context effectively while computation scales linearly with sequence length. Longer attention patterns, jumping to distant positions at regular intervals, allow information to propagate across the full context despite the local window constraint. The combination of local sliding windows with long-jump connections creates an effective receptive field that grows with the number of layers, enabling deep representations to incorporate information from distant contexts without quadratic computational scaling.

TheFlash Attention algorithm, developed by Dao and colleagues, represents perhaps the most significant practical advancement in transformer efficiency. Rather than computing attention in the traditional manner, which requires materializing the full $n \times n$ attention matrix in high-bandwidth memory, Flash Attention reorganizes computation to process attention in blocks that fit in fast GPU SRAM while using DRAM only for storing final outputs. The algorithm exploits the fact that attention can be computed in a numerically stable online manner, progressively updating results as blocks are processed. This approach yields attention computation that is memory-efficient without approximation—exact attention is computed, but without the memory overhead that typically makes long sequences infeasible. Modern implementations achieve 2-4x speedup on standard attention operations while reducing memory usage by an order of magnitude, enabling context windows of 128K tokens and beyond on available hardware.

Understanding pre-training objectives