The Pre-Transformer Era: The Quiet Before the Storm

November 12, 2016. The coffee line at the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP) in Austin, Texas, stretched past the ballroom doors, a sea of Patagonia vests and laptop stickers bearing the logos of FAIR, Google Brain, and CMU. Dr. Elena Marquez, a postdoc at the University of Washington who had spent the past 18 months building a legal document summarization system based on stacked bi-LSTMs, leaned against the wall next to her colleague, scrolling through her experiment logs with a sigh. “Another long-document test failed,” she said, tapping her screen. “The model can’t remember the contract clause from 300 words back. It just makes up a summary based on the last 50 words, like it always does.” Her colleague nodded, stirring sugar into his espresso. “I tried adding layer normalization, residual connections, even a 12-layer LSTM last week. The gradient just vanishes after 100 timesteps. I think we’re hitting a wall.”

Marquez’s frustration was not an outlier. It was the universal experience of the NLP research community in 2016, a year when the field was dominated by recurrent neural network (RNN) and long short-term memory (LSTM) architectures that had held sway for nearly two decades, but were now clearly hitting a fundamental performance ceiling. To understand why the Transformer revolution was not just an incremental improvement but a paradigm shift, we first have to reconstruct the quiet desperation of the pre-transformer era: the technical constraints that bound researchers, the failed workarounds that consumed years of labor, and the small group of engineers who were starting to imagine a world without recurrent loops.

The Recurrent Hegemony: How RNNs Came to Rule NLP

To appreciate the stakes of 2016, we have to rewind to the early 2010s, when RNNs and their LSTM variant displaced earlier statistical NLP methods (n-gram language models, support vector machines, and conditional random fields) to become the default architecture for almost every sequence modeling task. The appeal was obvious: unlike static feedforward networks, RNNs maintained a “hidden state” that updated with each new input token, theoretically allowing them to capture temporal dependencies across entire sequences. The LSTM, proposed by Hochreiter and Schmidhuber in 1997, added a gating mechanism to control the flow of information into and out of the hidden state, solving the early vanishing gradient problem that had crippled vanilla RNNs for short sequences.

By 2016, the standard NLP pipeline was built on LSTMs: bidirectional LSTMs for part-of-speech tagging and named entity recognition, stacked LSTMs with attention for machine translation and summarization, and LSTM-based language models for speech recognition and dialogue systems. A 2016 survey of ACL (Association for Computational Linguistics) conference papers found that 92% of all NLP research used RNN/LSTM architectures, with the remaining 8% split between CNNs, traditional statistical methods, and niche approaches like memory networks[^1]. The recurrent paradigm was not just popular—it was considered the only viable way to model sequential data, a belief so entrenched that many researchers considered the idea of a non-recurrent sequence model to be fundamentally flawed.

> 侧边栏：为什么顺序很重要？ > The core assumption underlying recurrent models is that sequential data has inherent order dependency: the meaning of a token depends on the tokens that come before it. For example, “the cat chased the mouse” and “the mouse chased the cat” have completely different meanings, so a model must process tokens in order to capture this dependency. For years, this assumption was considered unassailable—until the Transformer proved that order could be encoded without sequential processing.

The Cracks in the Foundation: Core Technical Limitations

For all their popularity, RNNs and LSTMs suffered from three fundamental, unsolved flaws that grew more severe as researchers tried to scale them to longer sequences and larger datasets. These flaws were not minor implementation bugs—they were inherent to the recurrent architecture itself, and no amount of hyperparameter tuning or architectural tinkering could fully fix them.

1. The Vanishing (and Exploding) Gradient Problem

The most well-documented flaw of recurrent models was the vanishing gradient problem, which made it nearly impossible to learn long-range dependencies across sequences longer than ~50–100 tokens. To understand why, we have to look at the math of backpropagation through time (BPTT), the algorithm used to train RNNs.

The core recurrence relation of a vanilla RNN can be written as: $$h_t = \sigma(W_{hh}h_{t-1} + W_{xh}x_t + b_h)$$ where $h_t$ is the hidden state at timestep $t$, $x_t$ is the input at timestep $t$, $W_{hh}$ and $W_{xh}$ are weight matrices, $b_h$ is the bias term, and $\sigma$ is the activation function (typically tanh or ReLU).

During backpropagation, the gradient of the loss $L$ with respect to the recurrent weight matrix $W_{hh}$ is computed as: $$\frac{\partial L}{\partial W_{hh}} = \sum_{t=1}^T \frac{\partial L}{\partial h_t} \left( \prod_{k=1}^{t-1} \frac{\partial h_k}{\partial h_{k-1}} \right) \frac{\partial h_1}{\partial W_{hh}}$$

The critical term here is the product of Jacobian matrices $\prod_{k=1}^{

$$\frac{\partial L}{\partial W_{hh}} = \sum_{t=1}^T \frac{\partial L}{\partial h_t} \left( \prod_{k=1}^{t-1} \frac{\partial h_k}{\partial h_{k-1}} \right) \frac{\partial h_1}{\partial W_{hh}}$$

The critical term here is the product of Jacobian matrices $\prod_{k=1}^{t-1} \frac{\partial h_k}{\partial h_{k-1}}$, which represents the gradient flow backward through each timestep of the sequence. For a vanilla RNN with tanh activation, the Jacobian $\frac{\partial h_k}{\partial h_{k-1}}$ has spectral norm (largest singular value) bounded between 0 and 1 for most weight initializations. As the sequence length $T$ grows, this product decays exponentially toward zero: for a sequence of length 100, the gradient from the final timestep is typically $10^{-10}$ to $10^{-15}$ times smaller than the gradient from the first timestep, effectively erasing all signal from the early parts of the sequence before it can update the weights. LSTMs mitigated this problem slightly by introducing a constant error carousel (the cell state $c_t$) that could propagate gradients unchanged across timesteps, but even LSTMs suffered from gradient decay for sequences longer than ~200 tokens, as the gating mechanism still required multiplicative interactions that attenuated signal over very long horizons.

> 侧边栏：梯度消失的直观类比 > Imagine passing a note through a chain of 100 people, where each person is only allowed to whisper 90% of what they hear to the next person. By the time the note reaches the 100th person, only 0.0027% of the original content remains—effectively indistinguishable from noise. Recurrent gradient flow works exactly the same way: each timestep attenuates the gradient by a factor of ~0.9, so after 100 steps, the signal is almost entirely lost.

The exploding gradient problem, the less frequently discussed but equally destructive counterpart to vanishing gradients, occurred when the spectral norm of the Jacobian product exceeded 1, causing gradients to grow exponentially and produce unstable, chaotic weight updates. While gradient clipping (a hack introduced by Pascanu et al. in 2013) could mitigate exploding gradients, it did nothing for vanishing gradients, and the two problems often alternated across layers and timesteps, making training long recurrent models a constant exercise in firefighting rather than systematic improvement. For Elena Marquez and her contemporaries, this meant that every experiment with longer sequences required hours of hyperparameter tuning just to get the model to converge, with no guarantee that the final performance would be better than a model trained on truncated sequences.

The P