MLOps 基础与生产鸿沟 / MLOps Foundations & The Production Gap

MLOps 基础与生产鸿沟

> MLOps Foundations & The Production Gap

1.1 鸿沟：被"最后一公里"吞噬的 87%

A 2020 Gartner survey reported that only 14% of ML projects had reached production; two years later the same analyst firm raised the number to ~48%, yet two-thirds of the remainder still fall into a "stuck-in-PoC" limbo. The picture is consistent across vendors. A 2022 Anaconda "State of Data Science" survey places the median time from first notebook to first production deployment at 14 months, while Microsoft's own internal telemetry, leaked in a 2023 paper, puts *model deployment abandonment* at 87% for projects led by small teams. If you, as an experienced practitioner, have ever felt that your neural network "worked perfectly" in the Jupyter notebook but somehow died the moment it crossed the CDN boundary, this chapter is for you.

The technical debt metaphor, originally coined by Ward Cunningham in 1992 for object-oriented code, was extended by Google engineers Sculley et al. in their landmark 2015 paper *"Hidden Technical Debt in Machine Learning Systems"* (NeurIPS). They showed that in a typical ML system, the model code itself (the small rectangle people tend to inspect) accounts for less than 5% of the total surface area. The remaining 95% is a sprawl of data pipelines, feature transforms, configuration files, serving infra, monitoring hooks, and feedback loops. Most of what kills an ML project is *not* a bad accuracy number; it is a missing breadcrumb that prevents you from reconstructing which data, which commit, which seed, and which infra image produced that accuracy number.

> Expert Insight 1.1 — Model Code is the Tip of the Iceberg > > A common cognitive error among researchers is to equate "the model" with "the product." In production, what you ship is the closure of the following equation: > > $$\text{Behavior}_{\text{prod}} \approx f(\text{model\_weights}, \text{feature\_defs}, \text{inference\_env}, \text{input\_schema}, \text{latency\_budget}, \text{threshold})$$ > > Touching any one variable while leaving the others pinned produces a system you can neither reason about nor debug. MLOps is precisely the discipline that makes such closed-loop reasoning cheap.

Throughout this chapter we will use three reusable constructs:

1. The Seven-Layer Technical Debt Model, a taxonomy I'll formalize into `TDML-7` (Section 1.3). 2. The CAMS Reframing, which transplants the DevOps culture of *Culture, Automation, Measurement, Sharing* onto the ML-specific substrate (Section 1.4). 3. The MLOps Maturity Ladder (Level 0-4), which lets us score any team against the same rubric (Section 1.6).

We close with a worked walkthrough of an e-commerce recommender (Section 1.7), a Canvas artifact (Section 1.8), and the three pillars that will recur in every later chapter (Section 1.9).

---

1.2 Why a "Normal" Software Mindset Fails Before It Even Starts

Before we taxonomize the debt, we need a sharper diagnosis. A classical microservice, when built well, has the property that changing inputs does not change its semantics: a correctly deployed billing engine with version 3.1.4 deployed yesterday will produce the same receipt for the same basket tomorrow, modulo configuration. An ML system *does not have this property*. Its semantics are a function of two time-dependent, partially-observable variables: the training data distribution $P_{\text{train}}(X, Y)$ and the incoming traffic distribution $P_{\text{serve}}(X)$. When $P_{\text{serve}} \neq P_{\text{train}}$ we say the model has *drifted*; the CI green checkmark from last week tells us nothing about the AUC of tomorrow's traffic.

This produces four classes of failure that have no clean analog in DevOps:

| Failure class | DevOps appearance | ML-specific manifestation | |---|---|---| | Silent regression | Lint/type-check fail in CI | Offline AUC unchanged, online CTR drops 30% | | Replay infeasibility | Reproducible binary build | Same commit, different GPU driver ⇒ different model | | Configuration blast radius | Feature flag flip | Threshold $\tau$ change ⇒ equity compliance breach | | Data-laundering | Secrets in repo | Test set leaks via shared shuffle seed |

Definition 1.1 — ML-induced Silent Regression. An apparently "working" model whose production behavior diverges from its training-time validation behavior due to data, code, or environment drift invisible to CI. The static CI gate is *necessary but not sufficient*.

A second, subtler issue is cost asymmetry of reversal. In DevOps, a bad deploy can typically be rolled back by reverting a Docker image; the *blast radius* is bounded by request timeouts and circuit breakers. In ML, by contrast, you sometimes discover a problem only when a downstream metric (e.g., retention, ad revenue, fraud loss) shifts days later, by which time the model has already *shaped* the very data you would train on next. Rolling back becomes a Causal Inference problem, not a feature-flag problem. We will return to this in Chapter 6 when we discuss champion-challenger design and counterfactual evaluation.

> Expert Insight 1.2 — Two Clocks in Production > > A reliable way to recognize ML debt during design review: ask the team whether they can answer all four of these questions *with a number, today*: > > 1. What was $P_{\text{serve}}(X)$ yesterday morning at 02:00 UTC? > 2. What is the 95th-percentile of the prediction latency over the last 24 hours? > 3. Which model version served the request that triggered the latest PagerDuty alert? > 4. What is the *predicted* AUC on the next 7 days if the current training cadence holds? > > A team that cannot answer even two of the four is, definitionally, at Maturity Level 0 or 1 — and the path forward is not "a better model" but a better production substrate.

---

1.3 The Seven-Layer Technical Debt Model (`TDML-7`)

We extend Sculley et al.'s original nine-debt classification into a layered model more suitable for engineering triage. Each layer *depends* on the layers below it; debt in a lower layer can undermine even a clean upper layer. The order, from substrate to surface, is deliberate.

```mermaid flowchart TD L1[L1: Data Dependency<br/>data drift, label leakage, schema breakage] L2[L2: Model Staleness<br/>concept drift, retrain cadence] L3[L3: Configuration Explosion<br/>feature flags, thresholds, env vars] L4[L4: Pipeline Glue<br/>cron jobs, shell scripts, ad-hoc DAGs] L5[L5: Feature-Model Versioning<br/>sklearn 0.24 vs 1.4 compat] L6[L6: Monitoring Blind Spots<br/>missing telemetry, no SLOs] L7[L7: Feedback-Loop Absence<br/>no prod-data to retrain pipeline] L1 --> L2 --> L3 --> L4 --> L5 --> L6 --> L7 ```

Layer 1 — Data Dependency. The first commitment your model makes in production is to an *implicit schema* formed by every column it touches. If upstream emits a new null in `user_agent`, your serving graph can degrade in three different ways (default fill, dtype cast, NaN propagation) and you have no test that distinguishes them. The mitigation is a contract-driven upstream using tools such as Great Expectations, TFX's `Schema`, or Pandera; we will dive into this in Chapter 3.

Layer 2 — Model Staleness / Rot. Even on a stable data stream, the world changes. *Concept drift* means $P(Y \mid X)$ shifts (fraudsters invent new patterns). *Data drift* means $P(X)$ shifts while $P(Y \mid X)$ holds (mobile users browse more at night). The robust answer is not "always retrain" but a sliding-window monitoring function

$$\Delta(t) = D_{\text{KL}}\!\left(\hat{P}_{\text{recent}}(X) \parallel \hat{P}_{\text{ref}}(X)\right),$$

with $\Delta(t) > \tau$ triggering a retrain. Choosing $\tau$ requires cross-team coordination that pure researchers often skip.

Layer 3 — Configuration Explosion. Each model ships with 30-300 *external* knobs: thresholds, calibration maps, encoding maps, fairness thresholds, fall-back class. The DevOps answ