数据的饥渴：当AI遇上了天花板 / The Hunger for Data: When AI Hits the Ceiling
数据的饥渴：当 AI 遇上了天花板
The Hunger for Data: When AI Hits the Ceiling
---
1.1 一个流传在山景城的传闻 / A Rumor from Mountain View
2023 年秋，OpenAI 内部发生了一件至今仍在硅谷私下流传的事。
那天深夜，几位研究员盯着不断攀升的 GPU 集群利用率曲线，发现了一个令人不安的事实：他们的模型损失函数（loss function）已经连续三周没有明显下降。换句话说，无论他们再喂入多少 token，模型都不再变聪明了。
最初的解释是超参数调优的问题。但当他们把模型架构、训练步长、学习率全部排查一遍后，答案浮出了水面——他们已经把互联网上几乎所有高质量的英文文本用完了。
In the fall of 2023, something happened inside OpenAI that has been whispered about in Silicon Valley ever since.
Late one night, a few researchers stared at the ever-rising GPU utilization curve and discovered something unsettling: their model's loss function had not meaningfully decreased for three consecutive weeks. No matter how many tokens they fed it, the model simply refused to get smarter.
The first explanation was a hyperparameter issue. But after exhaustively checking the architecture, training steps, and learning rate, the answer surfaced — they had used up essentially all the high-quality English text on the internet.
这不是孤例。Anthropic 的工程师在同一时期也公开承认，他们用来训练 Claude 2 的"图书语料"几乎耗尽了所有可公开获取的英文书籍。DeepMind 的一位研究主管在一次闭门会议上说得更直接："我们在 2024 年之前，会把整个互联网的'好文本'读完一遍；2026 年之前，会读完两遍。"
This was not an isolated case. Anthropic's engineers admitted during the same period that the "book corpus" used to train Claude 2 had nearly exhausted all publicly accessible English-language books. A research director at DeepMind put it even more bluntly in a closed-door meeting: "We will read through all the 'good text' on the internet once before 2024, and twice before 2026."
这就是"数据墙"（Data Wall）——本书第一个核心概念的诞生时刻。
This is the birth moment of the first core concept of this book: the "Data Wall."
---
1.2 水库模型：理解数据饥渴的第一把钥匙 / The Reservoir Model: Your First Key to Understanding Data Hunger
要理解数据为什么会成为瓶颈，我们先建立一个直观的比喻。
假设地球上所有可用的公开文本数据，是一个巨大的水库。
To understand why data becomes a bottleneck, let's first build an intuitive metaphor.
Imagine that all usable public text data on Earth is one enormous reservoir.
几十年来，这个水库的水位是稳定的。我们每年从河流（互联网新生成的内容）中取水，从湖泊（数字化的旧书、档案）中抽水，从冰川（被遗忘但仍可被检索的角落）中凿冰。AI 公司只要打开水龙头，水就来。
For decades, the water level in this reservoir has been stable. Every year we draw water from rivers (newly generated internet content), pump water from lakes (digitized old books, archives), and chisel ice from glaciers (forgotten but still retrievable corners). AI companies only need to turn on the faucet, and water comes.
但 2023 年前后，三件事同时发生了：
But around 2023, three things happened simultaneously:
1. 水龙头开得比河流补得快 — 模型对数据的需求呈指数级增长，而互联网新内容的生成接近线性。 2. 法律禁止了部分湖泊的抽水 — GDPR、CCPA、中国《数据安全法》等法规让大量数据变得不可直接使用。 3. 水开始变浑 — 互联网新生成内容的平均质量正在下降，AI 自己生成的内容又回流进数据集。
1. The faucet was opened faster than the rivers could refill — Model demand for data grew exponentially, while new internet content was generated roughly linearly. 2. The law prohibited pumping from some lakes — Regulations like GDPR, CCPA, and China's Data Security Law made vast amounts of data legally inaccessible. 3. The water was getting murky — The average quality of newly generated internet content was declining, and AI-generated content was flowing back into datasets.
让我把这三个趋势放在一张图里。
Let me put these three trends in one chart.
```mermaid flowchart TD A[公开数据水库 Public Data Reservoir] --> B[河流补给 River Replenishment<br/>互联网新内容 New Web Content] A --> C[湖泊 Lake<br/>数字化档案 Digitized Archives] A --> D[冰川 Glacier<br/>历史文献 Historical Records] E[AI 模型需求 AI Model Demand<br/>指数级增长 Exponential Growth] --> F[数据消耗 Data Consumption] F --> A G[隐私法规 Privacy Regulations<br/>GDPR, CCPA, 数据安全法] --> C H[数据质量衰减 Quality Degradation<br/>AI 生成内容回流 AI-Generated Reflux] --> B I[合成数据 Synthetic Data<br/>新水源 New Water Source] --> A ```
这张图描绘了 2023 年前后数据生态的真实结构：水库在被快速抽干，而补给的速度远远跟不上消耗的速度。唯一能够稳定注入水库的新水源，是"合成数据"——这也是本书第 3 章到第 10 章要展开的核心议题。
This chart depicts the real structure of the data ecosystem around 2023: the reservoir was being rapidly drained, while the rate of replenishment was far below the rate of consumption. The only new water source that can stably flow into the reservoir is "synthetic data" — which is the core topic of Chapters 3 through 10.
---
1.3 模型规模 vs 数据供给：一场不对称的赛跑 / Model Scale vs Data Supply: An Asymmetric Race
让我们把第一面墙放大来看——模型规模与数据供给的不对称增长。
Let's zoom in on the first wall — the asymmetric growth between model scale and data supply.
1.3.1 缩放定律的甜蜜期 / The Sweet Period of Scaling Laws
2020 年，OpenAI 发表了一篇里程碑式的论文：Scaling Laws for Language Models。这篇论文揭示了一个让整个行业狂喜的规律——只要等比例地增加模型参数、数据量、计算量，语言模型的能力就会可预测地提升。
In 2020, OpenAI published a landmark paper: Scaling Laws for Language Models. The paper revealed a rule that made the entire industry ecstatic — as long as model parameters, data volume, and compute were increased proportionally, language model capabilities would improve predictably.
数学上，这个关系可以近似表示为：
Mathematically, this relationship can be approximately expressed as:
$$L(N, D) \approx \left(\frac{N_c}{N}\right)^{\alpha_N} + \left(\frac{D_c}{D}\right)^{\alpha_D} + L_\infty$$
其中 $L$ 是损失（越低越好），$N$ 是参数量，$D$ 是数据量，$\alpha$ 是缩放指数，$L_\infty$ 是不可约损失。
Where $L$ is the loss (lower is better), $N$ is the parameter count, $D$ is the data volume, $\alpha$ is the scaling exponent, and $L_\infty$ is the irreducible loss.
这意味着一个诱人的前景：想要更强的 AI？训练更大的模型，喂更多的数据就行。
This meant a tantalizing prospect: want a stronger AI? Just train a bigger model and feed it more data.
2018 年的 GPT-1（1.17 亿参数）用了约 5GB 文本。 2019 年的 GPT-2（15 亿参数）用了约 40GB。 2020 年的 GPT-3（1750 亿参数）用了约 570GB。 2023 年的 GPT-4（参数未公开，估计 1 万亿+）用了估计数万亿 token。
GPT-1 in 2018 (117 million parameters) used about 5GB of text. GPT-2 in 2019 (1.5 billion parameters) used about 40GB. GPT-3 in 2020 (175 billion parameters) used about 570GB. GPT-4 in 2023 (parameters undisclosed, estimated 1 trillion+) used an estimated several trillion tokens.
4 年时间，参数增长了 100 倍，数据量增长了 1000 倍。
In 4 years, parameters grew 100x, and data volume grew 1000x.
1.3.2 数据增长的真相 / The Truth About Data Growth
但故事还有另一面。
But there's another side to the story.
研究机构 Epoch AI 在 2022 年的一项独立分析显示：高质量、公开发布、且未被污染的英文文本总量，大约只有 7 到 9 万亿 token。 这个数字听起来很大，但它有一个致命的问题——它基本是固定的。
An independent analysis by research organization Epoch AI in 2022 showed: the total amount of high-quality, publicly published, uncontaminated English text is roughly 7 to 9 trillion tokens. This number sounds large, but it has a fatal problem — it is essentially fixed.
相比之下，GPT-4 训练时消耗的 token 量级，已经接近或超过了这个上限。
In comparison, the token consumption during GPT-4's training already approached or exceeded this upper limit.
这就像是一个池塘里只有 1000 条鱼，但你用 5000 张网去捕——捕上来的鱼数量必然不如预期。
This is like having only 1000 fish in a pond but using 5000 nets to catch them — the catch will inevitably fall short of expectations.
更扎心的事实是：新数据的增长是线性的。 人类每年产生的新文本量虽然庞大，但相对于 AI 训练消耗的指数级增长，差距只会越来越大。
The more painful fact is: new data growth is linear. Although the amount of new text humans produce each year is huge, the gap with the exponential growth of AI training consumption will only widen.
1.3.3 当缩放定律撞墙 / When the Scaling Law Hits the Wall
2024 年，几家头部 AI 实验室相继报告了一个共同现象：他们的下一代模型预训练阶段，损失曲线的下降速度明显放缓。 这不是个别现象，而是整个行业第一次集体撞上"数据墙"。
In 2024, several leading AI labs successively reported a common phenomenon: in the pre-training stage of their next-generation models, the rate of decline of the loss curve slowed noticeably. This was not an isolated case — it was the industry's first collective crash into the "data wall."
Ilya Sutskever（OpenAI 联合创始人）在一次公开演讲中说了一句被反复引用的话：
Ilya Sutskever (OpenAI co-founder) said something repeatedly quoted in a public talk:
> "We have reached 'peak data.' The era of simply scaling up is over. We need new ideas."
> "我们已经达到了'数据峰值'。单纯扩大规模的年代结束了。我们需要新的想法。"
这句话标志着 AI 行业从"暴力美学"（brute force scaling）转向"精细工艺"（craftsmanship）的新阶段。
This remark marks the AI industry's transition from "brute force scaling" to a new phase of "fine craftsmanship."
---