AI治理的范式转变：从模型训练到系统治理 / The Paradigm Shift: From Model Training to System Governance
第 1 章 AI治理的范式转变：从模型训练到系统治理
Chapter 1: The Paradigm Shift — From Model Training to System Governance
> 本章核心问题 / Core Question: > 当模型从工具变成决策主体，传统工程实践为何不再足够？AI 治理的本质新范式是什么？ > *When models evolve from tools into decision-makers, why do traditional engineering practices fall short? What is the essential new paradigm of AI governance?*
---
1.1 醒钟：三个真实治理失败案例的根因分析
1.1 A Wake-Up Call — Root Cause Analysis of Three Real Governance Failures
2024 年 3 月，欧洲银行业管理局（EBA）对一家头部消费金融科技公司处以 480 万欧元罚款，原因是其信用风险评分模型自 2021 年 6 月上线后，从未触发任何再训练或漂移评估流程，在底层人群分布已显著右移的前提下，仍按 24 个月前的 score cut-off 放款，直接违反了欧盟《人工智能法案》（EU AI Act）第 9 条对"高风险 AI 系统持续监测义务"的强制性要求。这不是孤立事件。同期，Gartner 发布的调查显示，超过 64% 的企业 AI 系统在生产环境中至少经历过一次"治理失效"事件——即模型仍在运行，但围绕它的身份、决策、证据、成本、风险五个维度的治理机制出现实质性断裂。
*In March 2024, the European Banking Authority (EBA) fined a top consumer fintech €4.8 million because its credit risk scoring model — deployed since June 2021 — had never triggered any retraining or drift evaluation workflow. Despite significant rightward shifts in the underlying population distribution, it continued issuing loans using score cut-offs from 24 months earlier, directly violating Article 9 of the EU AI Act's mandatory "continuous monitoring obligation" for high-risk AI systems. This was not an isolated incident. A Gartner survey published the same period showed that over 64% of enterprise AI systems had experienced at least one "governance failure" event in production — where the model kept running, but the governance mechanisms across five dimensions (identity, decision, evidence, cost, risk) had substantively broken.*
在展开框架之前，让我们先用三个案例刻画"治理失败"的真实形态。请注意，这三个案例分别代表三类典型根因：权限边界缺失、审计日志断裂、生命周期终止机制缺位。这三类根因在样本上将反复出现——它们构成了本章小节 1.2 论证"传统 MLOps 不够"的实证基础。
*Before we lay out the framework, let us use three cases to characterize the true shape of "governance failure." Note that these three cases each represent a distinct class of root cause: missing permission boundaries, broken audit logs, and absent lifecycle termination mechanisms. These three root causes will recur repeatedly across samples — they form the empirical foundation for the argument in Section 1.2 that "traditional MLOps is insufficient."*
---
案例一 / Case 1: 权限边界缺失 — 信贷模型的"幽灵超管"
*Case 1: Missing Permission Boundaries — The "Ghost Superuser" of a Credit Model*
某消费金融公司的反欺诈模型托管在 Kubernetes 上，团队使用 Argo CD 管理 GitOps 工作流。一次内部审计发现：一名已离职的算法工程师的 IAM 凭证在过去 11 个月内，仍可访问生产模型卡（Model Card）的修改接口。该接口允许调用者直接绕过审批流程，将模型版本从 `v1.4-stable` 回滚到 `v1.3-experimental`。审计追溯发现，2023 年 7 月 12 日凌晨 3:47，曾有人通过该凭证将反欺诈模型的判断阈值从 0.72 修改为 0.68，直接导致次日的误拒率上升 4.7 个百分点，引发社交媒体上的客诉激增。
*A consumer finance company's anti-fraud model was hosted on Kubernetes, with the team using Argo CD for GitOps workflow management. An internal audit discovered that the IAM credentials of an algorithm engineer who had left the company could still access the production Model Card modification interface for 11 months. The interface allowed callers to bypass approval workflows and roll back the model version from `v1.4-stable` to `v1.3-experimental`. Audit tracing revealed that at 3:47 AM on July 12, 2023, someone using these credentials changed the anti-fraud model decision threshold from 0.72 to 0.68, directly causing a 4.7 percentage point rise in false rejection rate the next day and a spike of customer complaints on social media.*
根因解剖 / Root Cause Dissection:
1. 身份治理缺位 (Absence of Identity Governance): 模型没有"归属于某个责任实体"的强约束。一个模型的修改权限通常被授予"模型所有人"（Model Owner），但该字段在模型卡中是自由文本（"@alex-t"），没有指向企业 IdP（Identity Provider）中的真实账号。 2. 权限粒度过粗 (Coarse Permission Granularity): Argo CD 角色绑定到"项目"（Project）而非"模型卡"，工程师获得一个项目的写入权限即等同于获得该项目下所有模型卡的生产写入权限。 3. 离职自动失效链路断裂 (Broken Offboarding Automation): HR 系统标记离职后，IT 部门关闭的是 VPN 与 Jira 账号，但未同步吊销模型平台的 IAM 凭证。
*The identity of a model had no strong constraint to "belong to a responsible entity." The Model Owner field was free text ("@alex-t") rather than pointing to a real account in the enterprise IdP. Permission granularity was at the Project level in Argo CD, not the Model Card level — gaining write access to one project meant write access to every Model Card under it. The offboarding automation only closed VPN and Jira accounts but failed to revoke the IAM credentials on the model platform.*
---
案例二 / Case 2: 审计日志断裂 — 医疗影像 AI 的"沉默裁决"
*Case 2: Broken Audit Logs — The "Silent Adjudication" of a Medical Imaging AI*
某三甲医院部署了肺结节筛查 AI 系统，与放射科 PACS（Picture Archiving and Communication System）双向集成。2024 年 1 月，一名患者的家属以"漏诊延误治疗"为由提起诉讼，医院需要在 30 天内提交完整的 AI 决策证据链。
*A tertiary hospital deployed a lung nodule screening AI system, bidirectionally integrated with the radiology PACS. In January 2024, a patient's family filed a lawsuit alleging "missed diagnosis leading to delayed treatment," and the hospital needed to submit a complete AI decision evidence chain within 30 days.*
法务部门联系算法团队时发现：模型虽然记录了"预测分数"，但以下信息完全缺失：
- 输入特征（CT 切片）是否被预处理脚本修改过（哈希校验不存在）
- 模型推理时使用的是 `v2.1.0` 还是 `v2.1.1`（版本字段在模型卡中但与服务实例不绑定）
- 与本次推理并行的 A/B 实验配置（实验系统日志与推理日志分散在两个时钟源上）
- 触发本次推理的临床医生身份（系统日志显示是 "system"）
*When the legal department contacted the algorithm team, they found that although the model recorded a "prediction score," the following information was completely missing: whether input features (CT slices) had been modified by the preprocessing script (no hash verification existed); whether the model used `v2.1.0` or `v2.1.1` for inference (version field was in the Model Card but not bound to the service instance); the parallel A/B experiment configuration for this inference (experiment system logs and inference logs were on two separate clock sources); the identity of the clinician who triggered this inference (the system log showed "system").*
根因解剖 / Root Cause Dissection:
1. 证据链缺失 (Missing Evidence Chain): 模型推理日志只保留了"输出"，未保留"输入 + 上下文 + 配置 + 操作者"。这是 MLOps 中典型的"只 monitor 模型指标，不 log 数据沿革"的盲区。 2. 跨系统时钟漂移 (Cross-System Clock Drift): 实验系统与推理服务使用各自的 NTP 源，事后无法对齐"哪个实验配置影响了哪一次推理"。 3. 操作者身份字段被强制覆盖 (Operator Identity Field Forcefully Overwritten): 上游 API Gateway 在请求体中注入了 `user=system`，掩盖了真实临床医生身份。
*Inference logs kept only "output," not "input + context + configuration + operator." This is the typical MLOps blind spot of "monitoring model metrics but not logging data provenance." Cross-system clock drift meant the experiment system and inference service used separate NTP sources, making post-hoc alignment of "which experimental config influenced which inference" impossible. The operator identity field was forcefully overwritten by an upstream API Gateway injecting `user=system`.*
---
案例三 / Case 3: 生命周期终止机制缺位 — 推荐系统的"僵尸服务"
*Case 3: Absent Lifecycle Termination — The "Zombie Service" of a Recommendation System*
某电商平台在 2022 年关闭了旗下的"短视频"业务线，但底层支撑短视频推荐的 AI 服务并未下线。2023 年底，安全团队例行扫描时发现：该服务仍在 AWS 上以 8 核 16G 的 EC2 实例 7×24 小时运行，每月产生 1.7 万美元成本；其模型权重仍指向对象存储中一个被遗忘的 S3 桶；其推理端点暴露在公网，但由于上游业务流已停，无任何合法流量，因此仅有零星的扫描器流量与一次来自暗网的撞库尝试命中。
*An e-commerce platform shut down its "short video" business line in 2022, but the AI service underpinning short video recommendations was not decommissioned. At the end of 2023, a routine security scan found the service still running 7×24 on an 8-core 16G EC2 instance at $17,000/month; its model weights still pointed to a forgotten S3 bucket; its inference endpoint was publicly exposed, but with the upstream business flow stopped, there was no legitimate traffic — only sporadic scanner traffic and one credential-stuffing attempt from the dark web.*
根因解剖 / Root Cause Dissection:
1. 业务下线触发器未串联模型下线 (Business Shutdown Trigger Not Linked to Model Decommissioning): 业务关停流程仅通知财务、法务、IT 资产回收，未通知模型治理团队。 2. 缺乏"无主 AI 资产"巡检 (No "Ownerless AI Asset" Patrol): 内部 CMDB（Configuration Management Database）只跟踪 EC2/S3 资源，不跟踪"模型资产"。模型资产被遗忘在 12 个不同的内部平台中。 3. 暴露面持续存在 (Continued Attack Surface): 公网推理端点 14 个月未下线，期间累计收到 3.2 万次扫描请求。
*The business shutdown flow only notified Finance, Legal, and IT Asset Recovery — it did not no