Foundations of Natural Language Processing / 自然语言处理基础

自然语言处理基础

第一章：Foundations of Natural Language Processing

自然语言处理（Natural Language Processing, NLP）是一门让计算机理解、解释和生成人类语言的技术学科。作为人工智能最古老也最活跃的研究领域之一，NLP在过去七十余年间经历了从符号逻辑到统计学习，再到深度学习的深刻变革。理解这一演进历程，不仅是掌握现代语言模型的前提，更是洞察人工智能发展脉络的关键。本章将带领读者追溯NLP从规则系统到神经网络的技术演进，深入探讨词表示方法、分布语义学以及循环神经网络等核心概念，为后续章节介绍的Transformer架构奠定坚实的理论基础。

Natural Language Processing (NLP) is the discipline that enables computers to understand, interpret, and generate human language. As one of the oldest and most active research areas in artificial intelligence, NLP has undergone profound transformations over the past seven decades—from symbolic logic to statistical learning, and then to deep learning. Understanding this evolutionary trajectory is not only a prerequisite for mastering modern language models but also crucial for gaining insight into the broader development trajectory of artificial intelligence. This chapter takes readers through NLP's technological evolution from rule-based systems to neural networks, delving deeply into word representations, distributional semantics, and core concepts such as recurrent neural networks, laying a solid theoretical foundation for the Transformer architecture introduced in subsequent chapters.

规则系统时代：符号主义的兴衰

在20世纪50年代至80年代之间，NLP研究主要采用基于规则的方法。这种方法的核心理念是：语言可以被视为一套由有限规则支配的符号系统，只要能够精确地描述这些规则，计算机就能理解和生成自然语言。这一时期最具影响力的研究成果包括乔姆斯基（Noam Chomsky）的转换生成语法（Transformational-Generative Grammar）以及 ELIZA 等早期对话系统。

During the 1950s to 1980s, NLP research primarily employed rule-based approaches. The core idea of this methodology was that language could be viewed as a symbolic system governed by finite rules—if these rules could be precisely described, computers could understand and generate natural language. The most influential research during this period included Noam Chomsky's Transformational-Generative Grammar and early dialogue systems like ELIZA.

ELIZA系统由约瑟夫·维森鲍姆（Joseph Weizenbaum）于1966年在麻省理工学院开发，被认为是世界上第一个聊天机器人。ELIZA通过简单的模式匹配和替换规则模拟心理治疗师的对话风格。例如，当用户说"I need some help"时，系统可能将其转换为"Who says you need some help"并返回给用户。尽管ELIZA并不真正理解语言，但它展示了计算机程序模拟自然对话的可能性，这一概念至今仍是对话系统设计的核心之一。

The ELIZA system, developed by Joseph Weizenbaum at MIT in 1966, is considered the world's first chatbot. ELIZA simulated a psychotherapist's conversational style through simple pattern matching and substitution rules. For instance, when a user said "I need some help," the system might transform this into "Who says you need some help" and return it to the user. Although ELIZA did not truly understand language, it demonstrated the possibility of computer programs simulating natural conversation—a concept that remains central to dialogue system design today.

然而，规则系统面临根本性的局限性。首先，自然语言充满了例外、不一致和歧义，这些难以用明确的规则捕捉。其次，规则系统的开发需要语言学专家的密集参与，成本高昂且难以规模化。更重要的是，随着规则数量的增加，系统各组件之间的交互变得越来越复杂，导致难以维护和扩展。这种被称为"知识获取瓶颈"的问题，最终促使研究者转向统计方法。

However, rule-based systems faced fundamental limitations. First, natural language is replete with exceptions, inconsistencies, and ambiguities that are difficult to capture with explicit rules. Second, developing rule-based systems required intensive involvement from linguistic experts, making them costly and hard to scale. More importantly, as the number of rules increased, interactions among system components became increasingly complex, making maintenance and extension difficult. This problem, known as the "knowledge acquisition bottleneck," ultimately prompted researchers to shift toward statistical methods.

统计方法的崛起：数据驱动的语言建模

20世纪80年代末至90年代初，NLP领域发生了范式转换，研究者开始利用大规模语料库和统计学习技术来处理语言。这一转变的标志是1980年代末期马尔科姆·乔菲（Malcolm Joby）关于语料库语言学的工作，以及1990年代初期雅各布·洛希（Jacob Louser）提出的统计语言模型概念。统计NLP的核心假设是：语言的使用模式可以通过对大量文本数据的统计分析来发现和利用。

In the late 1980s to early 1990s, the NLP field underwent a paradigm shift, with researchers beginning to use large-scale corpora and statistical learning techniques to process language. This transformation was marked by Malcolm Joby's work on corpus linguistics in the late 1980s and Jacob Louser's concept of statistical language models in the early 1990s. The core hypothesis of statistical NLP was that language usage patterns could be discovered and exploited through statistical analysis of large amounts of textual data.

统计语言模型的基本思想是建立词的序列概率分布。设有一个词序列 $w_1, w_2, ..., w_n$，语言模型的目标是计算这个序列的概率 $P(w_1, w_2, ..., w_n)$。根据概率论的链式法则，这个联合概率可以分解为：

$$P(w_1, w_2, ..., w_n) = P(w_1) \cdot P(w_2|w_1) \cdot P(w_3|w_1, w_2) \cdot ... \cdot P(w_n|w_1, ..., w_{n-1})$$

Statistical language models establish probability distributions over sequences of words. Given a word sequence $w_1, w_2, ..., w_n$, the goal of a language model is to compute the probability of this sequence $P(w_1, w_2, ..., w_n)$. By the chain rule of probability, this joint probability can be decomposed as shown in the formula above.

然而，直接计算条件概率 $P(w_t|w_1, ..., w_{t-1})$ 面临严重的维度灾难问题。历史上下文的长度可能非常大，导致可能的词序列数量呈指数级增长。为解决这一问题，研究者引入了马尔可夫假设（Markov Assumption）：假设当前词的概率只依赖于前面的固定数量个词。如果只依赖于前 $n-1$ 个词，则称为 $n$-阶马尔可夫模型：

$$P(w_t|w_1, ..., w_{t-1}) \approx P(w_t|w_{t-n+1}, ..., w_{t-1})$$

However, directly computing the conditional probability $P(w_t|w_1, ..., w_{t-1})$ faces a severe curse of dimensionality problem. The length of historical context can be very large, causing the number of possible word sequences to grow exponentially. To address this, researchers introduced the Markov Assumption: the assumption that the probability of the current word depends only on a fixed number of preceding words. If it depends only on the preceding $n-1$ words, it is called an $n$-gram Markov model.

$n$-gram模型是统计NLP的基础工具。通过计算词序列在语料库中的出现频率，我们可以估计各种 $n$-gram的概率。例如，在一个三元语法模型（trigram model）中，我们需要计算 $P(w_t|w_{t-2}, w_{t-1})$，即给定前两个词时当前词的概率。这种方法简单有效，但存在数据稀疏问题——许多合理的词序列在训练语料中可能从未出现过，导致零概率问题。

$n$-gram models form the foundation of statistical NLP. By calculating the frequency of word sequences in corpora, we can estimate the probabilities of various $n$-grams. For example, in a trigram model, we need to compute $P(w_t|w_{t-2}, w_{t-1})$, the probability of the current word given the previous two words. This approach is simple and effective but suffers from data sparsity—the many reasonable word sequences that may never appear in training corpora lead to zero probability problems.

平滑技术（smoothing）被用来缓解数据稀疏问题。常见的平滑方法包括拉普拉斯平滑（Laplace Smoothing）、Kneser-Ney平滑和Katz回退等。这些技术通过将概率质量从高频事件重新分配给低频或未观察到的事件，提高了模型对未见数据的泛化能力。

Smoothing techniques are used to mitigate data sparsity. Common smoothing methods include Laplace Smoothing, Kneser-Ney Smoothing, and Katz Backoff. These techniques improve generalization to unseen data by redistributing probability mass from high-frequency events to low-frequency or unobserved events.

20世纪90年代至21世纪初，统计方法在NLP的各个任务上都取得了显著进展。词性标注（Part-of-Speech Tagging）、命名实体识别（Named Entity Recognition）、句法分析（Syntactic Parsing）等任务都采用了基于统计的学习方法。特别是约书亚·本吉奥（Yoshua Bengio）等人在2003年提出的神经语言模型（Neural Language Model），首次将神经网络引入语言建模，为后续的深度学习方法开辟了道路。

From the 1990s to the early 2000s, statistical methods achieved significant progress across various NLP tasks. Tasks such as Part-of-Speech Tagging, Named Entity Recognition, and Syntactic Parsing all adopted statistical learning approaches. Notably, Yoshua Bengio and colleagues proposed the Neural Language Model in 2003, which first introduced neural networks into language modeling, opening the path for subsequent deep learning approaches.

词表示：从离散符号到连续向量

语言模型的有效性很大程度上取决于如何表示语言中的词汇。词表示（Word Representation）是NLP中最基础也最关键的问题之一。在历史上，研究者尝试了多种词表示方法，从最初的独热编码到分布式表示，构成了现代词嵌入技术的基础。

The effectiveness of language models depends largely on how words in language are represented. Word representation is one of the most fundamental and critical issues in NLP. Throughout history, researchers have explored various word representation methods, from initial one-hot encoding to distributed representations, forming the foundation of modern word embedding techniques.

独热编码（One-Hot Encoding）是最直观的词表示方法。假设我们的词汇表包含 $V$ 个词，每个词被表示为一个 $V$ 维向量，其中只有一个位置为 1（表示该词），其余位置均为