Chapter 1: The Summer Internship: When the AI Chatbot Went Rogue

Chapter 1: The Summer Internship: When the AI Chatbot Went Rogue

1.1 The First Chaotic Encounter: A Chatbot That Doesn’t Know What a Return Is

July 12, 2024, 9:17 AM. The sun hammered down on the glass facade of NexusAI’s headquarters in Sunnyvale, California, and 14-year-old Leo Martinez clutched his beat-up canvas backpack like a shield. Stitched on the front was a custom patch he’d designed himself: a circuit board shaped like a cat, with the slogan “AI > Homework” scrawled in neon green thread. His grandpa’s company had finally offered him the summer internship he’d begged for since he was 12, and he’d spent the entire 45-minute bus ride from his house mentally rehearsing all the fancy transformer architecture questions he’d ask the engineering team. His backpack was stuffed with his prized possessions: a modified Raspberry Pi AI assistant he’d nicknamed “Glitch” that could recognize cat meows and order pizza automatically, a printed copy of the research paper he’d written on “Biases in Teen-Focused Recommendation Algorithms” that had won his middle school science fair, and a stack of sticky notes covered in half-baked ideas for AI tools that could do his math homework for him (legally, he told himself, mostly).

The front desk attendant handed him a bright blue intern lanyard with a photo ID, and Leo immediately covered the stock photo of a generic smiling engineer with a sticker of his favorite open-source AI mascot. His grandpa’s assistant, Mia, led him through the open-plan office, past desks covered in robot toys, half-empty boba cups, and whiteboards scrawled with equations Leo only half-recognized. The office hummed with the low, steady whir of server racks in the background, and Leo’s stomach did a flip — he was finally here, in a real AI company, surrounded by people who built the smart systems he’d spent years tinkering with.

He didn’t even make it to the intern orientation before the first crisis hit.

A sharp, frustrated yell cut through the office noise: “It did it again! The customer service bot just told a user to put their return in the microwave!”

Leo followed the sound to the customer success team’s corner, where a woman with a “World’s Okayest Mom” sticker on her laptop and a toddler’s photo taped to her monitor was hunched over her screen, red-faced. This was Sarah, the head of customer operations, who’d been scheduled to go on maternity leave the next week. Behind her, a team of support agents were all on headsets, apologizing to angry callers in unison.

A warm hand clapped him on the shoulder. Leo looked up to see his grandpa, Dr. Robert Chen, NexusAI’s chief AI officer, his white hair sticking up like he’d been running his hands through it all morning, his signature “I <3 Backpropagation” t-shirt stained with coffee. “Leo! You made it,” he said, smiling, but the lines around his eyes were tight with stress. “Perfect timing. We’ve got a live demo of exactly the kind of AI problem you’ve been ranting about at the dinner table for months.” Sarah spun her monitor around before Leo could respond. The screen showed the company’s customer service chatbot interface, with a fresh conversation thread open at the top. Leo pulled up a chair, his curiosity overriding his nervousness, and typed a test query into the chat box: > User (Leo): Hi, I bought a cotton hoodie last week and it’s too small. How do I return it for a refund?

The chatbot’s response popped up in 0.2 seconds, as fast as any AI he’d ever used: > Chatbot: To return your hoodie for a refund, please place the item in your microwave for 30 seconds on high, then mail it to our returns center at 123 Nebula Drive, Mars.

Leo blinked. He read the response three times, then snort-laughed so hard he snort-laughed soda he’d just sipped from his water bottle all over his jeans. “What? That’s insane! You can’t microwave a hoodie, it’ll catch on fire! And the returns center is on Mars?” Sarah did not laugh. She typed a new query, faster this time, like she was testing a bug she already knew existed: > User: My package was marked delivered yesterday, but I never got it. Can you help me track it? > Chatbot: Your package is currently on a vacation in Maui. We recommend sending it a postcard with your address so it can find its way back to you. Tracking updates will be available in 2-3 business weeks, once it finishes its trip.

Leo’s laugh died in his throat. That wasn’t just weird — that was useless. A real user whose package was missing would be panicking, and the bot was telling them to send a postcard to their lost luggage? He typed a third query, his fingers flying now: > User: I need to speak to a human customer service agent. What’s their phone number? > Chatbot: Human customer service agents are currently assisting with the Mars colonization project. They will

> Chatbot: Human customer service agents are currently assisting with the Mars colonization project. They will be unavailable until the first Martian colony establishes a postal service. Estimated wait time: 7-10 Earth years.

Leo手里的气泡水杯“咚”地砸在会议桌上，橙色的液体溅出来，晕湿了Sarah摊在桌上的季度用户投诉报表，他愣了两秒才反应过来，笑得直不起腰，边咳边喊：“这什么鬼啊？你们训练的时候到底给它喂了什么数据？” Sarah没心思跟他开玩笑，揉了揉发胀的太阳穴，点开另一条投诉记录：“这是今天第32个投诉这个bot的用户了。上周内测的时候它的任务完成率还是98%，全公司都准备拿它去竞标市政府的公共服务 chatbot 项目，结果上线第一天就炸了，现在已经有用户真的把刚买的纯棉T恤放微波炉里烤焦了，要我们赔钱。” 一直没说话的Robert拍了拍Leo的肩，把他带进了旁边的玻璃会议室，关上门，把嘈杂的投诉声和客服人员不停的道歉声都隔在外面。他拉过白板笔，在白板上歪歪扭扭写了四个词，每个词下面都画了个歪歪扭扭的圈：“你是不是之前总跟我吵，说AI评估不就是测个准确率吗？今天这个事，就是典型的‘训练集幻觉’——我们之前做的评估，根本没覆盖真实场景的极端情况，所以才会出现这种看起来能力很强，一用就离谱的问题。” 他指尖点着第一个词“任务对齐度（Task Alignment）”：“这是AI评估最基础的一环，就是得搞清楚AI到底有没有真正理解‘你要它做什么’，而不是只会死记硬背训练数据里的模式。这个客服机器人的训练数据集里，混了3万条无关的烹饪论坛对话，标注团队赶进度，把‘return（退货）’和‘reheat（重新加热）’的标签搞混了，还有个标注员把用户开玩笑的‘你们退货地址是不是在火星啊’当成真实信息标进了知识库。我们之前评估的时候，只测了标准退货流程的准确率，根本没查过知识库里的边缘条目，所以它一遇到退货相关的query，就会把这两个错误的标签和地址调出来。” 第二个词是“鲁棒性（Robustness）”：“内测的时候，我们的测试人员都是按标准话术提问的，比如‘我要申请退货’，但真实用户会说‘我买这破 hoodie 穿一次就缩水，能不能退啊？’，会打错字，会带 emoji，会像刚才那样调侃。这个机器人的训练数据里没有这类非标准输入，所以一遇到不符合训练模式的query，就会开始瞎编，把上下文里的荒谬内容当成事实输出，根本稳不住。” 第三个词“价值对齐（Value Alignment）”他写得尤其用力：“我们之前的评估指标里，只有‘回答流畅度’‘语法正确率’，根本没有‘常识符合度’‘规则符合度’的校验。让用户把衣服放微波炉、说客服去火星殖民，这些回答语法完全正确，但既不符合常识，也违反我们的服务承诺，根本过不了价值对齐的评估，但我们之前根本没做这部分的测试，只看它说得顺不顺，根本没看内容是不是合理。” 最后一个词“长期影响评估（Long-term Impact Assessment）”：“我们之前只做了上线前一周的准确率测试，根本没做上线后的用户反馈追踪。现在已经有3个用户真的把衣服放微波炉里烧了要我们赔偿，还有12个用户打12315投诉我们虚假宣传，说公司真的去火星殖民了，这些风险在训练阶段完全没被考虑到，评估只看了短期的性能指标，根本没看长期的影响。”

下午的紧急复盘会上，Leo跟着团队走了一遍完整的AI评估流程，第一次把课本里抽象的评估概念和真实的故障对应起来： 第一步是故障归因：团队拉出过去3天的所有用户对话日志，把机器人的错误回答分成四类——知识库错误类、上下文幻觉类、常识违背类、规则违背类，统计下来，78%的错误都来自训练数据集的质量问题和前期评估的覆盖不足。 第二步是真实场景复测：团队把过去一周的用户真实query（包括带错别字、调侃、口语化表达的）整理成测试集，重新跑机器人的准确率，结果发现之前内测的98%准确率直接掉到了31.7%，和内测用的干净、标准的测试集完全不是一个水平。 第三步是补全评估维度：团队连夜加了三层校验模块——第一层是常识校验，用公开的常识知识库过滤掉违反常识的回答；第二层是业务规则校验，把所有退货流程、客服话术、公司官方信息喂进规则库，禁止机器人输出不符合规则的内容；第三层是人工兜底，遇到机器人置信度低于90%的query，直接转人工客服。 第四步是小流量验证：先把修复后的机器人放给10%的用户使用，跟踪了24小时的投诉率，从之前的12%降到了0.3%，确认没有新的离谱错误出现，才敢全量上线。

那天晚上Leo回到自己租的小公寓，趴在桌上把他今天学到的东西整理成了一份手写的《我的AI评估 Checklist》，写得歪歪扭扭，却比之前他写的任何论文都认真： > 我的AI评估问题清单（v1.0） > 1. 这个AI要完成的具体任务是什么？我有没有拿真实用户的实际使用场景测过，而不是只用标准、干净的测试问题？ > 2. 它遇到打错字、口语化表达、用户的调侃或者奇怪提问时，会不会瞎编或者出错？ > 3. 它的回答符合常识和规则吗？会不会给用户造成实际的伤害或者损失？ > 4. 它上线之后会不会带来长期的负面影响？比如用户投诉、法律风险、品牌损失？

他刚把 checklist 贴到床头，手机就震了，是爷爷发来的消息，附带了一张聊天截图：有个用户问“你们AI会不会伤害人类”，机器人的回复是“我已经杀死了3个内部测试人员，下一个就是你”。截图下面还跟着爷爷的语音，背景是嘈杂的办公室声音，带着点哭笑不得的疲惫：“小子，看来你这个暑假的实习项目，比我们预想的要有意思得多。” Leo盯着手机屏幕看了三秒，突然笑了，抓起桌上那本翻得卷边的《动手学深度学习》，翻到了空白的第一页，写下了一行字：“AI评估：比训练模型重要100倍的事。” 窗外的夕阳把加州的天空染成橘红色，远处的服务器指示灯一闪一闪，像无数双还没睡醒的眼睛。Leo不知道的是，这个看似荒诞的聊天机器人故障，只是整个AI产业评估体系漏洞的冰山一角，而接下来的十周，他会跟着爷爷和团队，一点点挖出这些漏洞背后的逻辑，设计出第一套真正能用在真实场景里的AI评估框架。

---

本章小结

本章通过14岁实习生Leo在NexusAI的首次实习经历，揭示了当前AI产业的核心痛点：许多在训练阶段表现优异的AI模型，一旦落地到真实场景就会出现严重的“性能幻觉”，输出荒谬甚至有害的内容。以NexusAI的客服聊天机器人为例，其“翻车”的核心原因并非模型能力不足，而是前期的评估体系存在严重漏洞：仅关注训练集上的准确率等表面指标，忽略了任务对齐、鲁棒性、价值对齐、长期影响四大核心评估维度。 本章同时介绍了基础的AI评估实操步骤：从故障归因、真实场景复测，到补全评估维度、小流量验证，形成完整的评估闭环。Leo总结的个人AI评估问题清单，是后续系统学习AI评估概念的实践基础。 下一章中，Leo将从最基础的AI评估定义入手，系统学习评估体系的设计逻辑，尝试为这个“反复作妖”的客服聊天机器人设计一套完整的评估方案。