崩溃的周五夜晚：当单体应用撞上流量洪峰 / The Friday Night Crash: When a Monolith Meets the Traffic Tsunami
第一章：崩溃的周五夜晚 —— 当单体应用撞上流量洪峰
Chapter 1: The Friday Night Crash — When a Monolith Meets the Traffic Tsunami
> *"在分布式系统中，所有故障都不是技术问题，而是时间问题。"* > > *"In distributed systems, every failure is a question of when, not whether."* > > —— 来自一位不愿透露姓名的 SRE
---
1.1 开场：生日蜡烛与红色警报
Opening: Birthday Candles and Red Alerts
故事要从一个周五的傍晚讲起。
This story begins on a Friday evening.
那是 2023 年 11 月 10 日，距离所谓的"光棍节"还有整整二十四小时。对于杭州这座电商之城而言，这是一年里最重要的前夜 —— 所有的服务器、所有的运维、所有和数字有关的灵魂，都在为即将到来的零点倒计时屏息凝神。
It was November 10, 2023, exactly twenty-four hours before what the marketing calendar calls "Singles' Day." For Hangzhou, this city of e-commerce, it was the most important night of the year — every server, every operator, every digital soul was holding its breath in anticipation of the midnight countdown.
林薇（Lin Wei）已经在 TechFlow 的订单系统团队摸爬滚打了七年。从最初写 JSP 的初级工程师，到如今统管着三十多人、近百个微服务（是的，百个微服务 —— 但这并非一开始就是百个）的技术经理，她见过太多的"大促"。但今年，她罕见地没有在公司值守，而是坐在自家客厅的餐桌前，看着六岁的女儿朵朵（Duo Duo）认真地吹灭七根彩色蜡烛。
Lin Wei had been working at TechFlow's order system team for seven years. From her early days writing JSP pages as a junior engineer to her current role as a technical manager overseeing a team of over thirty people and nearly a hundred microservices (yes, a hundred microservices — though it didn't start that way), she had seen too many "big promotions." But this year, unusually, she wasn't stationed at the company. Instead, she sat at the dining table in her living room, watching her six-year-old daughter Duo Duo carefully blow out seven colorful candles.
"妈妈许愿！"朵朵拍了拍小手。
"Mommy, make a wish!" Duo Duo clapped her small hands.
林薇闭上眼睛。她心里默念的不是什么宏大的人生理想，而是一串非常具体的、工程师式的祈愿 —— *今晚，订单系统不要再出幺蛾子*。
Lin Wei closed her eyes. What she silently recited wasn't some grand life aspiration, but a very specific, engineer-style prayer — *tonight, please don't let the order system go haywire*.
烛光摇曳。丈夫在厨房里洗盘子。窗外下着小雨。这本该是一个普通得不能再普通的周五夜晚。
The candlelight flickered. Her husband was washing dishes in the kitchen. Rain pattered softly against the window. This should have been the most ordinary of Friday nights.
但 21:47:23，命运决定不让她如愿。
But at 21:47:23, fate decided otherwise.
林薇的手机震动了三下。第一下是钉钉的告警音，第二下是企业微信的@提醒，第三下 —— 是她私人的那个 Slack 频道，那个只有她和组长们加入的"oncall-救火"群，正在疯狂刷屏：
Her phone buzzed three times. The first was a DingTalk alert sound, the second a WeChat Work @mention, and the third — her private Slack channel, the "oncall-firefighting" group that only she and team leads joined — was scrolling frantically:
``` [21:47:21] [CRITICAL] order-core P99 latency: 15200ms (>SLA 3000ms) [21:47:23] [CRITICAL] db-pool-01 exhausted: 200/200 connections used [21:47:25] [CRITICAL] tomcat-thread-pool blocked: 800/800 threads RUNNING [21:47:28] [FATAL] checkout-api 5xx error rate: 47.3% [21:47:31] [FATAL] user-facing banner: 服务暂时不可用 (Service Temporarily Unavailable) ```
林薇的瞳孔在那一瞬间收缩了。
Lin Wei's pupils contracted in that instant.
"妈，怎么了？"朵朵抬起头，黑葡萄一样的眼睛看着她。
"Mommy, what's wrong?" Duo Duo raised her head, looking at her with grape-like black eyes.
"没什么，宝贝。妈妈工作上有点急事，要回公司一趟。"她已经在穿外套了。
"It's nothing, baby. Mommy's got something urgent at work, I need to go to the office." She was already putting on her coat.
二十分钟后，她已经在出租车后座，双手捧着笔记本，连上了公司的 VPN。监控大屏在她的浏览器里展开，像一张突然起了火的航拍地图：所有的曲线都飘红了。
Twenty minutes later, she was in the back seat of a taxi, laptop in hand, connected to the company's VPN. The monitoring dashboard unfurled in her browser like an aerial photo of a city suddenly on fire: every curve had turned red.
而这，仅仅是开始。
And this was only the beginning.
---
1.2 雪崩：一座数字城市的崩塌
The Avalanche: The Collapse of a Digital City
要理解那一夜发生了什么，我们必须先理解 TechFlow 在那个时间点的系统长什么样。
To understand what happened that night, we must first understand what TechFlow's system looked like at that moment.
2023 年的 TechFlow，已经是一家估值 30 亿美金的电商独角兽。他们的核心业务 —— 也就是这天晚上正在燃烧的这套系统 —— 依然是 2017 年那位早已离职的首席架构师留下的"遗产"：一个名为 `monolith-order` 的 Spring Boot 单体应用，部署在 16 台物理机上，使用 MySQL 5.7 主从架构，前端由 Nginx 负载均衡。
TechFlow in 2023 was already a $3 billion valued e-commerce unicorn. Their core business — the system currently on fire that night — was still the "legacy" left behind by a chief architect who had long since departed in 2017: a Spring Boot monolith application named `monolith-order`, deployed on 16 physical machines, using a MySQL 5.7 master-slave architecture, with Nginx for front-end load balancing.
没错，2023 年了，他们还在用单体应用跑着一年 GMV 超过 800 亿人民币的核心交易链路。
Yes, in 2023, they were still running a monolith application handling a core transaction chain with an annual GMV exceeding 80 billion RMB.
林薇曾在无数次会议上提出过微服务改造。但董事会的口径出奇一致 ——"现在还没出问题，为什么要动它？"
Lin Wei had proposed microservice transformations in countless meetings. But the board's answer was remarkably consistent — "It's not broken yet, why fix it?"
直到今晚。
Until tonight.
让我们看看那一夜究竟发生了什么。这是一个经典的"雪崩（Avalanche）"过程，几乎可以写进任何分布式系统教科书的反面案例。
Let's see what actually happened that night. This is a classic "avalanche" process, one that could almost be written into the negative case studies of any distributed systems textbook.
第一阶段：流量的第一波冲击
Phase One: The First Wave of Traffic
21:30 左右，随着各大直播平台开始预热，大量的用户涌入 TechFlow 的 App。系统的每秒请求数（RPS, Requests Per Second）从平峰的 1,200 稳步攀升到 4,500。
Around 21:30, as major livestream platforms began their warm-up sessions, massive numbers of users flooded TechFlow's app. The system's requests per second (RPS) climbed steadily from an off-peak 1,200 to 4,500.
订单系统的 P99 延迟（在 99% 的请求都能在该时间内完成的延迟指标）从基线的 180ms 缓慢爬升到了 420ms。这并不稀奇 —— 系统本身就预留了一定的弹性。MySQL 的连接池从 200 个连接里用了 85 个，Tomcat 的线程池也才占用 60%。
The order system's P99 latency (the latency under which 99% of requests complete) climbed slowly from its baseline 180ms to 420ms. This wasn't unusual — the system itself had built-in elasticity. MySQL's connection pool was using 85 out of 200 connections, and Tomcat's thread pool was only at 60% utilization.
但林薇的直觉告诉她 —— 这不是简单的流量增长。她注意到一个细节：订单创建接口的延迟在涨，但订单查询接口的延迟涨得更凶。
But Lin Wei's intuition told her — this wasn't simply traffic growth. She noticed a detail: the order creation API's latency was rising, but the order query API's latency was rising even more aggressively.
这意味着什么？
What did this mean?
这意味着 —— 商品详情页和购物车页面在大量调用订单查询，而订单查询现在和订单写入共享着同一套 MySQL 实例、同一份连接池、同一组 Tomcat 线程。当订单写入因为流量上涨开始变慢时，订单查询也遭了殃。
This meant — the product detail pages and shopping cart pages were calling the order query API extensively, and the order query now shared the same MySQL instance, the same connection pool, the same set of Tomcat threads with the order writes. When the order writes began to slow down due to traffic increase, the order queries suffered too.
这就是单体应用最经典的"资源争抢（Resource Contention）"问题：你没办法只扩展一个资源池，因为所有功能都跑在同一个进程里。
This is the most classic "Resource Contention" problem in monolith applications: you can't scale just one resource pool, because all functionality runs in the same process.
第二阶段：连接池的临界点
Phase Two: The Connection Pool Tipping Point
21:47 左右，真正的转折点出现了 —— 某位著名的头部主播在 A 直播间突然开始带货 TechFlow 的明星单品。
Around 21:47, the real turning point appeared — a famous top-tier influencer suddenly began promoting TechFlow's star product in Livestream Room A.
瞬间，10 万个用户点击了"立即购买"按钮。
In an instant, 100,000 users clicked the "Buy Now" button.
RPS 在 30 秒内从 4,500 飙升到 18,000。这是设计容量的近 4 倍。
RPS surged from 4,500 to 18,000 within 30 seconds. This was nearly 4 times the design capacity.
订单写入请求大量涌入。在 MySQL 5.7 的默认事务隔离级别（RR, Repeatable Read）下，每一笔订单创建都需要拿到排他锁（X-lock），而所有的订单修改 —— 包括库存扣减、优惠券核销、积分变动 —— 都在同一个事务里。数据库的锁等待队列（InnoDB lock wait queue）瞬间堆积到了 8,000 多个。
Massive order write requests flooded in. Under MySQL 5.7's default transaction isolation level (RR, Repeatable Read), every order creation needed to acquire exclusive locks (X-locks), and all order modifications — including inventory deduction, coupon verification, points changes — were happening in the same transaction. The database's lock wait queue (InnoDB lock wait queue) insta