Prologue: The World Before the Transformer

Imagine, for a moment, the year is 2016. You are sitting at your desk, waiting for a machine translation service to convert a paragraph from English to French. The sentence is complex, with multiple clauses and a nuanced idiom near the beginning. You hit ‘translate’ and watch a progress bar crawl across the screen. Seconds tick by. Finally, the result appears. The idiom is rendered literally, losing its cultural meaning. The subject of the final clause, which was clearly stated in the original, has been replaced by a vague pronoun. The translation is technically grammatical but feels… hollow. The machine has read your sentence word by word, meticulously, sequentially, and by the time it reached the end, it had, in a very real sense, forgotten the beginning.

This was the world before the Transformer. A world governed by the tyranny of the sequence.

Our journey begins not with a solution, but with a profound and deeply felt limitation. To appreciate the seismic shift brought about by the 2017 paper “Attention Is All You Need,” we must first immerse ourselves in the cognitive and computational landscape it shattered. This is the era of Recurrent Neural Networks (RNNs) and their more robust offspring, Long Short-Term Memory networks (LSTMs). For over a decade, these architectures were the undisputed sovereigns of Natural Language Processing (NLP), the engines powering everything from smartphone keyboards to early machine translation systems. Their reign was characterized by a beautiful, intuitive, and ultimately constraining philosophy: language is fundamentally sequential, and therefore, intelligence must process it one step at a time, carrying a whisper of the past into the present.

This chapter is our ‘before’ picture. We will paint it in detail, not as a story of failure, but as a testament to human ingenuity in working within a tight constraint. We will explore the elegant mechanics of recurrence, the desperate engineering required to combat its flaws, and the palpable tension between the human mind’s parallel, holistic grasp of meaning and the machine’s plodding, left-to-right march. We will meet the key players at Google who were quietly assembling the pieces of a new paradigm, sensing that the sequential bottleneck was not a feature but a bug. By the end of this prologue, you will not just understand what the Transformer replaced; you will feel the intellectual claustrophobia of the old world and understand why the revolution was not just welcome, but inevitable. The stage is set. The actors are in place. The old guard is about to be overthrown.

---

The Sequential Imperative: The Core Metaphor of Pre-2017 NLP

At the heart of the pre-Transformer paradigm lay a powerful, almost seductive metaphor: the mind as a river, and language as a stream of tokens flowing past a single point of observation. An RNN embodies this metaphor literally. At each time step *t*, it consumes a single input token *x_t* (a word, or more commonly, a dense word embedding), and updates its internal state, or “hidden state” *h_t*. This hidden state is the network’s working memory—its sole vessel for carrying information forward. The output *y_t* at that step is a function of the current input and this accumulated memory. The process is inherently serial; you cannot compute the state at *t=5* without first having computed the states at *t=1* through *t=4*.

```mermaid graph LR subgraph "RNN Unfolding in Time" direction LR A[x1] --> B[h1] B --> C[y1] B --> D[h2] D --> E[y2] D --> F[h3] F --> G[y3] F --> H[...] end ```

This sequential dependency is both its strength and its Achilles' heel. It provides a straightforward way to model temporal dynamics—the idea that the meaning of a word can depend on what came before. The recurrence creates a chain of influence, a narrative thread. But it also creates a computational and informational bottleneck.

Computationally, the serial nature means the steps cannot be parallelized across a modern GPU’s thousands of cores. Processing a sentence of length *n* requires *n* sequential operations. For long documents, this becomes prohibitively slow. Training, which requires many passes over vast corpora, becomes a marathon of sequential steps, not a sprint of parallel computation.

Informationally, the hidden state is a narrow channel. As the sequence length grows, the network must compress the entire history of the sequence up to that point into a fixed-size vector. This is the famous vanishing gradient problem (and its cousin, the exploding gradient problem). During training, the error signal must be propagated backward through this chain of states to adjust the earlier weights. After many steps, this signal tends to either vanish to zero (meaning early layers don’t learn) or explode into instability. The network becomes effectively short-sighted, struggling to learn dependencies that span more than a few dozen tokens. It forgets.

The LSTM, introduced by Hochreiter & Schmidhuber in 1997 and popularized in the early 2010s, was the brilliant, knightly response to this forgetfulness. It didn’t discard the sequential metaphor; it fortified the memory vessel.

---

The Ingenious Fortress: LSTMs and the War Against Forgetting

The LSTM’s central innovation is the cell state (*c_t*), a separate, privileged information highway that runs through the entire sequence with minimal interference. It is modulated by three (later four) gates—forget gate, input gate, and output gate—which carefully regulate what information to discard from the cell state, what new information to store, and what to expose as the hidden state.

```mermaid graph TD subgraph "LSTM Cell State Flow" direction LR A[C_t-1] --> F[Forget Gate] B[X_t] --> I[Input Gate] I --> C[Candidate] F --> D[+] C --> D D --> E[C_t] E --> O[Output Gate] O --> H[H_t] end ```

Mathematically, the core operations (using a simplified formulation) are:

1. Forget Gate: $f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$ Decides what percentage of the previous cell state $c_{t-1}$ to keep. 2. Input Gate & Candidate: $i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$ $\tilde{c}_t = \tanh(W_c \cdot [h_{t-1}, x_t] + b_c)$ Decides what new information to consider and generates a candidate update. 3. Cell State Update: $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$ The forget gate’s output is multiplied element-wise with the old state, and the input gate’s output is multiplied with the candidate. They are added to form the new cell state. This additive nature is key—it allows gradients to flow relatively unchanged through the cell state path, mitigating vanishing gradients. 4. Output & Hidden State: $o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$ $h_t = o_t \odot \tanh(c_t)$ The hidden state (the output for the next step) is a filtered view of the cell state.

This architecture was a masterpiece of engineering. For years, LSTMs and their cousin, the Gated Recurrent Unit (GRU), were the workhorses of NLP. They powered the first wave of practical machine translation (e.g., Google’s Neural Machine Translation system in 2016), speech recognition, and text generation. They *felt* like they had memory. You could prime them with a paragraph and they would generate coherent follow-on text for a while.

Yet, the fundamental constraints remained. The sequential computation bottleneck was absolute. You could not process the 10th word until the 9th was finished. This crippled training speed and made it difficult to scale to the massive datasets that would soon become available. More subtly, the sequential inductive bias was still there. The model’s primary mechanism for building understanding was to walk step-by-step through the text, updating i