神经网络语言模型的演进：从N元模型到Transformer

语言模型（Language Model）是自然语言处理领域的核心概念之一，其本质是对自然语言序列进行概率建模的系统。更精确地说，一个语言模型的任务是计算一个给定序列 $P(w_1, w_2, \ldots, w_n)$ 的概率，或者预测下一个词的条件概率 $P(w_t | w_1, w_2, \ldots, w_{t-1})$。这种能力看似简单，却是几乎所有自然语言处理任务的基础——从机器翻译到文本生成，从语音识别到问答系统，都依赖于对语言概率分布的准确建模。

The language model, at its core, is a probability distribution over sequences of words or tokens. Mathematically, we aim to compute either the joint probability $P(w_1, w_2, \ldots, w_n)$ of an entire sequence, or more commonly in modern approaches, the conditional probability $P(w_t | w_1, w_2, \ldots, w_{t-1})$ of the next token given all preceding tokens. This seemingly simple objective—capturing the statistical regularities of natural language—underpins virtually every NLP application, from machine translation to text generation, from speech recognition to question answering.

理解语言模型的技术演进，对于把握当前大语言模型（Large Language Models, LLMs）的本质至关重要。从20世纪80年代的N元语法（N-gram）模型，到21世纪初的前馈神经网络语言模型（Neural Network Language Model, NNLM），再到2013年的词向量表示方法，以及随后催生Transformer架构的注意力机制，这条演进路径上的每一个里程碑都解决了一个关键瓶颈，同时又带来了新的挑战。正是这些积累与突破，最终在2017年后催生了GPT、BERT等现代大语言模型，开启了人工智能的新纪元。

Understanding this technological evolution is essential for grasping the nature of modern large language models. The journey from N-gram models in the 1980s, through feedforward neural language models in the early 2000s, to word embeddings in 2013, and finally to the attention mechanism that gave birth to the Transformer architecture—each milestone solved a critical bottleneck while introducing new challenges. It is this accumulation of breakthroughs that ultimately gave rise to GPT, BERT, and other modern LLMs after 2017.

统计语言模型：N元语法的基石与局限

Statistical Language Models: N-gram Foundations and Limitations

N元语法模型的基本原理

The Fundamental Principles of N-gram Models

统计语言模型的核心思想可以追溯到信息论与概率论的结合。1948年，克劳德·香农（Claude Shannon）在其开创性论文《通信的数学理论》（A Mathematical Theory of Communication）中，首次提出用概率统计方法描述语言信息。他通过N元字符序列（character-level N-grams）的实验，证明了语言具有可计算的统计结构。这一理论为后来的统计语言模型奠定了基础。

The foundational idea of statistical language models traces back to the confluence of information theory and probability theory. In 1948, Claude Shannon's seminal paper "A Mathematical Theory of Communication" proposed using probabilistic methods to characterize linguistic information. Through experiments with N-gram sequences of characters, Shannon demonstrated that language possesses computable statistical structure—a finding that laid the groundwork for statistical language models.

N元语法模型的核心假设是：一个词的出现概率仅依赖于它前面固定的$N-1$个词。这一假设被称为马尔可夫假设（Markov Assumption），它将联合概率分解为条件概率的乘积：

The core assumption of N-gram models is that the probability of a word depends only on the preceding $N-1$ words. This is known as the Markov Assumption, which decomposes the joint probability into a product of conditional probabilities:

$$P(w_1^n) = P(w_1, w_2, \ldots, w_n) = \prod_{k=1}^{n} P(w_k | w_1^{k-1}) \approx \prod_{k=1}^{n} P(w_k | w_{k-N+1}^{k-1})$$

其中 $w_i^j$ 表示从第 $i$ 个词到第 $j$ 个词的序列。当 $N=1$ 时，模型退化为仅考虑词频的一元模型（Unigram）；$N=2$ 时是常用的二元模型（Bigram），只考虑前一个词的影响；$N=3$ 时是三元模型（Trigram），考虑前两个词的影响。更高的$N$值能够捕捉更丰富的上下文信息，但面临数据稀疏和计算复杂度呈指数增长的问题。

When $N=1$, the model reduces to a unigram that considers only word frequencies; when $N=2$, we have the commonly used bigram model that considers only the previous word; when $N=3$, we have the trigram considering the two preceding words. Larger values of $N$ can capture richer contextual information but face the twin challenges of data sparsity and exponential growth in computational complexity.

在实际应用中，选择合适的$N$值需要在模型表达能力与数据稀疏问题之间取得平衡。典型的选择范围是$N=2$到$N=5$：

In practice, choosing an appropriate $N$ involves balancing model expressiveness against data sparsity. The typical range is $N=2$ to $N=5$:

| N值 | 模型名称 | 特点 | 典型应用场景 | |-----|---------|------|-------------| | 1 | Unigram | 词频统计，无上下文依赖 | 简单文本统计、词袋模型 | | 2 | Bigram | 考虑前一个词 | 基础语音识别、输入法联想 | | 3 | Trigram | 考虑前两个词 | 传统机器翻译、拼写检查 | | 4-5 | Higher-order N-gram | 捕捉更长依赖 | 语音识别（带语言模型） |

平滑技术：解决数据稀疏问题

Smoothing Techniques: Addressing Data Sparsity

N元语法模型面临的一个根本问题是数据稀疏（Data Sparsity）。自然语言中词表规模庞大（通常在数万到数十万级别），而即使是三元模型，其可能的词序列组合数也达到$|V|^3$的量级（$|V|$为词表大小）。在实际语料中，大多数可能的N元组合从未出现过，导致零概率问题——这不仅影响概率估计的准确性，更会导致整个序列的概率为0。

A fundamental problem with N-gram models is data sparsity. The vocabulary in natural language is vast (typically tens to hundreds of thousands of words), and even trigram models face a combinatorial explosion of possible sequences, on the order of $|V|^3$ where $|V|$ is the vocabulary size. In actual corpora, most possible N-gram combinations never occur, leading to zero-probability problems that not only affect probability estimation accuracy but can render entire sequences' probabilities zero.

平滑技术（Smoothing）通过重新分配概率质量来解决这一问题。最基本的拉普拉斯平滑（Laplace Smoothing），也称为加一平滑（Add-one Smoothing），对所有计数加1：

Smoothing techniques address this by redistributing probability mass. The most basic approach is Laplace Smoothing, also known as Add-one Smoothing, which adds 1 to all counts:

$$P_{Laplace}(w_k | w_{k-N+1}^{k-1}) = \frac{c(w_{k-N+1}^{k-1}, w_k) + 1}{c(w_{k-N+1}^{k-1}) + |V|}$$

其中$c(\cdot)$表示N元组的计数，$|V|$是词表大小。加一平滑虽然简单，但在实际应用中往往过于激进，因为它将大量从未出现的组合赋予了相等的概率。

While simple to implement, Laplace smoothing is often too aggressive in practice, as it assigns equal probability to all unseen combinations.

更精细的平滑方法包括：

More sophisticated smoothing methods include:

Good-Turing平滑是信息论学者Good和Turing在二战期间为破解恩尼格玛密码而开发的，后来被引入语言建模。其核心思想是：观察出现$r$次的N元组，在未见数据中大约出现$r+1$次的N元组也有类似的行为。具体公式为：

Good-Turing smoothing was developed by Good and Turing during World War II for breaking the Enigma code, later adapted for language modeling. The core idea is that an N-gram observed $r$ times behaves similarly to an N-gram that appears $r+1$ times in unseen data. The formula is:

$$P_{GT}(w | context) = \frac{r^*(w, context)}{c(context)}$$

其中$r^*$是打折后的计数，基于出现频率为$r$的N元组数量进行重新估计。

Where $r^*$ is the discounted count, re-estimated based on the number of N-grams with frequency $r$.

Kneser-Ney平滑是1995年由Reinhard Kneser和Hermann Ney提出的，至今仍是最有效的N元语法平滑方法之一。它采用一种精妙的插值思想：低阶模型提供back-off概率，高阶模型提供主要概率估计，同时考虑上下文的多样性。其核心公式为：

Kneser-Ney smoothing, proposed by Reinhard Kneser and Hermann Ney in 1995, remains one of the most effective N-gram smoothing methods. It employs an elegant interpolation approach where lower-order models provide back-off probabilities while higher-order models provide the main probability estimates, considering the diversity of contexts:

$$P_{KN}(w_i | w_{i-N+1}^{i-1}) = \frac{\max(c(w_{i-N+1}^i) - d, 0)}{\max(c(w_{i-N+1}^{i-1}), 0)} + \lambda(w_{i-N+1}^{i-1}) \cdot P_{KN}(w_i | w_{i-N+2}^{i-1})$$

其中$d$是一个与计数相关的折扣参数，$\lambda$是归一化常数，确保概率和为1。

Where $d$ is a discount parameter related to the count, and $\lambda$ is a normalization constant ensuring probabilities sum to 1.

N元模型的实际应用与局限性

Practical Applications and Limitations of N-gram Models

尽管N元语法模型有明显的局限性，但它在特定场景下仍然具有实用价值。以Google的语音搜索为例，2010年前后其语言模型就是基于超过万亿词的大规模语料构建的N元模型，在服务器端运行内存占用达数百GB。这种大规模N元模型能够捕捉丰富的语言现象，在特定领域（如新闻、对话）表现出色。

Despite their limitations, N-gram models retain practical value in specific scenarios. Google's voice search around 2010 used N-gram models built on corpora exceeding a trillion words, consuming hundreds of gigabytes of server-side memory. Such large-scale N-gram models capture rich linguistic phenomena and perform well in specific domains like news and conversation.

然而，N元模型的根本缺陷是无法捕捉长距离依赖和复杂的语义关系。以句子"The book that I bought yesterday was very expensive and I put it on the shelf"为例，理解"it"指代"book"需要依赖长距离的语言理解能力，而N元模型最多只能捕捉局部的统计规律。

However, the fundamental flaw of N-gram models is their inability to capture long-range dependencies and complex semantic relationships. Consider the sentence "The book that I bought yesterday was very expensive and I put it on the shelf"—understanding that "it" refers to "book" requires long-range linguistic comprehension that N-gram models, limited to local statistical patterns, simply cannot achieve.

此外，N元模型无法处理未登录词（Out-of-Vocabulary, OOV）问题，无法进行词语