第一章 翻车之夜：当模型在排行榜上夺冠，却在用户面前崩溃 / Chapter 1 The Night of the Crash

第一章 翻车之夜：当模型在排行榜上夺冠，却在用户面前崩溃

Chapter 1 The Night of the Crash

凌晨两点十七分，林舟的手机震动了十七次。

凌晨 2:17, Lin Zhou's phone buzzed seventeen times.

他没有数错。每一个震动都对应着一条来自「极光智能」——他共同创办的 AI 公司——客户支持频道的告警消息。窗外是北京 CBD 沉睡的天际线，办公桌上摆着他们刚刚发布的旗舰大模型 Lumi-7B 的发布会纪念马克杯，杯身上的烫金字母在屏幕的冷光里反射出讽刺的弧度。/ He didn't miscount. Each buzz corresponded to an alert from the customer support channel of Aurora Intelligence, the AI company he co-founded. Outside the window slept the Beijing CBD skyline. On his desk sat a commemorative launch mug for Lumi-7B, their newly released flagship large model, the gold-foil letters on its surface glinting with a sarcastic arc in the cold screen light.

十七分钟前，全公司还在某个酒吧里庆祝 Lumi 在 MMLU（Massive Multitask Language Understanding，大规模多任务语言理解基准）上拿下 78.4 分——比上一代足足高出 9 个百分点，跃居同尺寸开源模型榜首。他们的 PR 团队连夜赶出了那条「国产之光」的微博长图，研发负责人端着香槟在人群中央讲了一个关于「长跑精神」的故事。/ Seventeen minutes earlier, the whole company had been celebrating at a bar: Lumi had scored 78.4 on MMLU—9 points higher than its predecessor, leaping to the top of open-source models of the same size. Their PR team had rushed out a "Light of China" long-form Weibo graphic, and the head of R&D had been telling a "long-distance running" story over champagne at the center of the crowd.

二十分钟后，第一个工单涌入。/ Twenty minutes later, the first support ticket poured in.

「你们的 AI 是疯了吗？我让它帮我写一封辞职信，它给我回了三段鼓励我坚持下去的鸡汤。/ "Is your AI insane? I asked it to write me a resignation letter and it replied with three paragraphs of chicken soup encouraging me to persevere."

「我跟它说『我今天被裁员了心情很差』，它反手给我推荐了一门『情绪管理付费课程』，还贴心地附上了优惠券。/ "I told it 'I got laid off today and I feel terrible,' and it shoved a paid 'emotional management course' recommendation at me, even kindly attaching a coupon code."

「让它帮我孩子做小学三年级数学题，第二步就错了，但它的语气像真理部发言人一样笃定。/ "I asked it to help my third-grader with math; the second step was wrong, but its tone was as certain as a Ministry of Truth spokesperson."

到凌晨三点，「Lumi 翻车」已经冲上了知乎热榜和微博热搜。/ By 3 a.m., "Lumi crashes" had climbed onto Zhihu and Weibo trending lists.

林舟看着告警面板上不断跳动的数字，第一次真切地感受到一种他此前只在教科书里读到过的现象：离线分数（offline score）与在线体验（online experience）之间的巨大鸿沟。/ Watching the numbers tick up on his alert dashboard, Lin Zhou felt for the first time a phenomenon he had only previously read about in textbooks: the yawning chasm between offline score and online experience.

这就是本书第一章要回答的核心问题——/ This is the central question this opening chapter will answer—

> 为什么一个在公开基准上表现优异的模型，部署到真实场景中却可能一败涂地？/ Why does a model that aces public benchmarks fail catastrophically in production?

---

1.1 那一夜，我们究竟做错了什么

1.1 What Exactly Did We Do Wrong That Night

事后复盘时，极光智能的事故复盘会（launch postmortem）持续了整整四个小时。/ In the post-mortem, Aurora Intelligence's launch postmortem lasted a full four hours.

会议桌的一侧坐着研发团队，每个人脸上都带着「我代码写得好好的，凭什么要认错」的委屈；另一侧是产品和运营，他们的 PPT 里密密麻麻地列着工单分类、情感倾向、用户流失曲线。/ On one side of the conference table sat the R&D team, every face wearing the grievance of "my code is fine, why should I take the blame"; on the other side were product and operations, whose PPTs were dense with ticket classification, sentiment analysis, and user churn curves.

林舟没有立刻发火。他让所有人先把观点放下，共同回答一个简单的问题：/ Lin Zhou didn't explode. He asked everyone to set aside their opinions and jointly answer one simple question:

> MMLU 78.4 分到底在度量什么？/ What exactly does an MMLU score of 78.4 measure?

会议室安静了三十秒。/ The meeting room was quiet for thirty seconds.

研发负责人先开口：「MMLU 衡量的是模型在 57 个学科、约 16,000 道选择题上的零样本（zero-shot）准确率。它是当前业界最权威的英文综合知识与推理基准之一。」/ The R&D lead spoke first: "MMLU measures a model's zero-shot accuracy on roughly 16,000 multiple-choice questions across 57 subjects. It's currently one of the most authoritative English comprehensive knowledge and reasoning benchmarks in the industry."

产品总监追问了一句：「那它是衡量『回答得正确』的能力，还是『回答得让人满意』的能力？」/ The product director asked: "Does it measure the ability to answer *correctly*, or the ability to answer *satisfactorily*?"

又是一阵沉默。/ Another silence.

「是『回答得正确』，」研发负责人艰难地承认，「而且是『在标准答案已知的情况下从四个选项里选出正确答案』。」/ "The former," the R&D lead conceded painfully, "and specifically 'pick the correct answer from four choices when the ground truth is known.'"

那一刻，整个团队意识到了问题的第一层：他们花了三个月、烧了数百万 GPU 小时去优化的那个数字，根本不是用户能感知的那个东西。/ At that moment, the whole team realized the first layer of the problem: that number they had spent three months and millions of GPU hours optimizing was fundamentally not the thing users could perceive.

1.1.1 公开基准的三大原罪

1.1.1 The Three Original Sins of Public Benchmarks

事后，他们把当晚暴露出的问题归纳为公开基准的「三大原罪」——/ Afterward, they summarized the problems exposed that night as the "three original sins" of public benchmarks—

原罪一：数据污染（Data Contamination）。/ Sin #1: Data Contamination.

MMLU 的题目虽然不是开源的，但它们的考点、所考察的概念、所涉及的事实——「美国宪法第一修正案保护哪些权利？」「微积分基本定理的两种形式分别是什么？」——都广泛地散落在互联网的每一个角落。训练一个 7B 参数的模型动辄要爬取数万亿 token（token，模型处理的最小文本单位，可以粗略理解为一个词或半个汉字）的网页，而 2023 年以后，几乎所有严肃的中文和英文网页都被用作训练语料。/ Although MMLU's exact questions are not open source, the concepts they test—"Which rights does the First Amendment protect?" "What are the two forms of the Fundamental Theorem of Calculus?"—are scattered everywhere on the internet. Training a 7B-parameter model typically involves crawling trillions of tokens (the smallest text unit a model processes, roughly a word or half a Chinese character) of web pages, and since 2023, virtually every serious Chinese and English web page has been used as training corpus.

这意味着什么？/ What does this mean?

这意味着模型在「考试」时看到的题目，可能在「上学」时就见过一模一样的「练习题」。/ It means that the questions a model "sees on the exam" may have been seen in identical form as "practice questions" during "school."

学术界有专门的方法去检测污染程度——比如 Membership Inference Attack（成员推断攻击）：训练一个分类器，看它能不能判断某道题是否在训练集里。如果模型「背题」了，它对原题的置信度（confidence，模型对答案把握程度的内部数值）会异常高，回答时的语言模式也会与新题显著不同。/ Academia has dedicated methods for detecting contamination—e.g., Membership Inference Attack: train a classifier to determine whether a given question was in the training set. If a model has "memorized the question," its confidence on the original will be abnormally high, and its linguistic pattern will differ significantly from new questions.

Lumi 的复盘报告显示：他们事后用 MIA 方法做了一次抽样检测，发现 MMLU 57 个学科里有 11 个学科存在显著污染迹象，其中包括「美国历史」「临床医学」和「法律」这三个用户投诉最密集的领域。/ Lumi's retrospective report showed: a sampled MIA test found 11 of MMLU's 57 subjects exhibited significant contamination signals, including "U.S. History," "Clinical Medicine," and "Law"—the three areas with the densest user complaints.

原罪二：维度单一（Single-Dimensional Reductionism）。/ Sin #2: Single-Dimensional Reductionism.

MMLU 给你一个数字：78.4。/ MMLU gives you a number: 78.4.

但用户关心的维度至少有以下七种：/ But users care about at least the following seven dimensions:

1. 正确性（Correctness）：答案对不对？/ *Correctness*: Is the answer right? 2. 事实性（Factuality）：答案里的事实陈述是否真实？/ *Factuality*: Are the factual claims in the answer true? 3. 有用性（Helpfulness）：是否解决了用户的问题？/ *Helpfulness*: Does it solve the user's problem? 4. 无害性（Harmlessness）：是否避免了伤害、偏见、毒害？/ *Harmlessness*: Does it avoid harm, bias, toxicity? 5. 流畅性（Fluency）：语言是否自然、得体？/ *Fluency*: Is the language natural and appropriate? 6. 一致性（Consistency）：相同问题不同时间回答是否一致？/ *Consistency*: Are answers to the same question consistent across time? 7. 风格契合（Style Fit）：语气、长度、格式是否符合用户期待？/ *Style Fit*: Does the tone, length, and format match user expectations?

一个选择题分数，把这七种维度压扁成一条线。/ A multiple-choice score flattens these seven dimensions into a single line.

更糟糕的是，它压扁的方式是有偏的：MMLU 几乎只测「正确性」和一点点「知识覆盖」，而「有用性」「无害性」「风格契合」这些用户最关心的维度完全不在它的评估范围之内。/ Worse still, the way it flattens is biased: MMLU measures almost only "correctness" and a little "knowledge coverage," while "helpfulness," "harmlessness," and "style fit"—the dimensions users care about most—are completely outside its evaluation scope.

这就是为什么 Lumi 能在 MMLU 上拿 78.4 分，却连一封辞职信都写不好——因为「写辞职信」这个任务考验的根本不是「知道什么是辞职信」，而是「能否在恰当的语境里产生恰当的共情」。/ This is why Lumi could score 78.4 on MMLU yet couldn't even write a proper