Foundations of Large Language Models

Foundations of Large Language Models

The effectiveness of any prompt engineering strategy ultimately traces back to the architectural decisions and training paradigms that define the underlying language model. When a practitioner crafts a carefully structured few-shot example or applies chain-of-thought reasoning, the success or failure of that technique depends on how information flows through millions of neural network parameters. Understanding these internals—how attention mechanisms propagate token relationships, how pretraining objectives shape grammatical and semantic knowledge, and how instruction tuning modifies base model behavior—enables practitioners to move beyond trial-and-error prompt modification toward principled strategy selection. This chapter establishes the technical foundation necessary to make those informed decisions.

Transformer Architecture and the Mechanics of Attention

The transformer architecture, introduced in the seminal "Attention Is All You Need" paper by Vaswani et al. (2017), fundamentally restructured how neural networks process sequential data. Unlike recurrent architectures that processed tokens sequentially, transformers enabled parallel computation across entire sequences, a property that proved essential for scaling to the billions of parameters characteristic of modern language models. Understanding this architecture requires tracing how individual tokens transform as they propagate through the network's layers.

The Self-Attention Mechanism

At the heart of the transformer lies the self-attention mechanism, which computes relationships between all positions in a sequence simultaneously. For each input position, attention calculates a weighted sum of value vectors, where the weights are determined by the compatibility between query and key vectors. Mathematically, for a sequence of $n$ tokens, the attention function produces:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

where $Q$, $K$, and $V$ are matrices derived from input embeddings through learned linear projections, and $d_k$ is the dimensionality of the key vectors. The scaling factor $\sqrt{d_k}$ prevents the dot products from growing too large in high dimensions, which would push the softmax function into regions with extremely small gradients.

Multi-head attention extends this mechanism by running multiple attention operations in parallel. Rather than performing a single attention function on $d_{model}$-dimensional inputs, the model projects the inputs into $h$ lower-dimensional subspaces, computes attention independently in each, and concatenates the results. Each head can attend to different aspects of the relationships between tokens—one might focus on syntactic dependencies, another on semantic similarity, and a third on positional proximity. The concatenated output then passes through a final linear projection:

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)W^O$$

where $\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)$.

The implications for prompt engineering are substantial. When a user provides a prompt containing examples, instructions, and a query, the attention mechanism determines how relevant each component is to each other. In a few-shot prompt, the examples compete for attention with the actual query, and the model's ability to recognize structural patterns in the examples depends on whether the attention heads learned to capture such relationships during pretraining. Chain-of-thought prompts succeed partly because intermediate reasoning steps create additional positions that the model can attend to when generating the final answer, effectively expanding the "working memory" available for complex reasoning.

```mermaid graph TD A[Input: "The cat sat on the mat"] --> B[Tokenization & Embedding] B --> C[Positional Encoding Addition] C --> D[Multi-Head Self-Attention Layer 1] D --> E[Add & Norm Layer 1] E --> F[Feed-Forward Network Layer 1] F --> G[Add & Norm Layer 1] G --> H[Multi-Head Self-Attention Layer N] H --> I[Add & Norm Layer N] I --> J[Feed-Forward Network Layer N] J --> K[Add & Norm Layer N] K --> L[Linear Projection] L --> M[Output Distribution]

N[Query] --> D O[Key] --> D P[Value] --> D ```

Positional Encodings and Sequence Understanding

The self-attention mechanism treats all positions symmetrically—there is no inherent notion of token order in the attention computation itself. To incorporate positional information, transformers add positional encodings to the input embeddings before the first layer. The original transformer paper employed sinusoidal encodings, using sine and cosine functions of different frequencies:

$$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$ $$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$

This design allows the model to learn to attend to relative positions because linear transformations of the positional encoding can represent fixed offsets. Subsequent work introduced learned positional embeddings, which the model optimizes directly during training, and more recent architectures like Rotary Position Embedding (RoPE) encode position through rotations in the attention computation rather than additive encodings.

The choice of positional encoding strategy affects how models handle long sequences and how they generalize to lengths not seen during training. Models with sinusoidal or RoPE encodings typically generalize better to longer sequences than those with learned embeddings, which may struggle with positions beyond their training context. For prompt engineering, this matters when crafting prompts that approach or exceed the model's training context length, as the positional encoding strategy determines how reliably the model can attend across distant prompt segments.

Layer Normalization and Residual Connections

Modern transformer architectures consistently employ layer normalization and residual (skip) connections around each sub-layer. The residual connection adds the input of a sub-layer to its output, creating a gradient highway that allows gradients to flow directly through the network during training. Layer normalization stabilizes training by normalizing activations across features rather than across the batch dimension:

$$\text{LayerNorm}(x) = \gamma \odot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$

where $\mu$ and $\sigma$ are the mean and standard deviation computed across the feature dimension, and $\gamma$ and $\beta$ are learned scaling and bias parameters.

These architectural elements have practical implications for understanding model behavior. Residual connections mean that each layer adds to the representation produced by previous layers, so information from early tokens can flow relatively unimpeded through the network to influence final predictions. Layer normalization ensures that activations remain in reasonable ranges, but its placement (pre-norm versus post-norm) affects training dynamics and potentially the characteristics of trained models.

Pretraining Objectives and Their Lasting Influence

The objectives used during pretraining fundamentally shape what knowledge and capabilities a language model develops. Two primary objectives dominate: causal (or autoregressive) language modeling, which predicts the next token given preceding context, and masked language modeling, which predicts randomly masked tokens given surrounding context. Understanding these objectives and their effects illuminates why different model architectures and training regimes produce models with different characteristics.

Causal Language Modeling

Causal language modeling trains models to predict the probability distribution over the next token given all preceding tokens. The training objective maximizes:

$$\mathcal{L}_{\text{CLM}} = \sum_{t=1}^{T} \log P(x_t | x