The Data-Centric Paradigm: Why the Pipeline Is the Product

Chapter 1: The Data-Centric Paradigm — Why the Pipeline Is the Product

第一章：以数据为中心的范式——为何流水线即产品

In late 2023, an open-source 70-billion-parameter language model, trained on roughly 0.5% of the tokens consumed by the largest frontier systems, matched frontier performance on a battery of standard reasoning benchmarks. The architecture was unremarkable: a standard transformer decoder with rotary positional embeddings, grouped-query attention, and a SwiGLU feed-forward block. Every one of these components had been published, replicated, and refined in the open literature over the preceding thirty-six months. What was different, and what the rest of this book takes as its starting premise, was almost entirely on the data side: aggressive deduplication at the document, paragraph, and sentence level; a hand-curated mixture over high-signal mathematical and code corpora; quality classifiers trained on human preference data; a synthetic question-generation pipeline that targeted the specific reasoning failures observed during evaluation; and a held-out evaluation harness that treated contamination as a first-class concern. The result was a model whose loss curve and downstream behavior were clearly a function not of what its architecture could in principle express, but of what its training corpus could in practice teach it.

This is not an isolated anecdote. It is the leading edge of a broader pattern that has been visible, in retrospect, since the Chinchilla inflection point of 2022. The empirical scaling laws that Kaplan et al. published in 2020 described a world in which loss fell smoothly as a power law in parameters, tokens, and compute, and in which the exponents were such that one could spend additional compute more or less freely on either dimension and expect a similar return. Hoffmann et al.'s Chinchilla correction showed that the exponents were not symmetric: at a fixed compute budget, the optimal model was much smaller and trained on much more data than Kaplan had recommended, and the penalty for under-training on data was severe. By 2023, multiple independent labs had demonstrated that the same correction applied to the post-Chinchilla regime itself: at very large compute budgets, the marginal return on additional parameters continued to shrink relative to the marginal return on additional *high-quality* data, and once a model was sufficiently over-parameterized, additional capacity without additional signal simply memorized more of the same distribution. The bottleneck had moved.

This chapter makes the case that, in the present regime, data engineering is the dominant determinant of model capability and the dominant determinant of long-term defensibility for AI products. The argument proceeds in four steps. We first show that the pre-training scaling laws have saturated along the parameter axis, and that the new bottleneck is data quality and coverage (Section 1.1). We then trace the intellectual lineage of the data-centric view, from hand-engineered features through representation learning to Andrew Ng's 2021 call to arms and the modern data-platform stack (Section 1.2). We define the term *data engine* and contrast it, property by property, with a static dataset (Section 1.3). We close with a detailed case study of a recommendation system that migrated from a frozen offline training corpus to a continuous data flywheel and observed a 14% lift in monthly active users (Section 1.4). The chapter ends by setting up the formal theory of data quality that will occupy Chapter 2.

A word on the audience. This book assumes the reader is comfortable with the standard deep-learning stack — transformers, optimizers, evaluation methodology — and with the basic vocabulary of production systems — feature stores, streaming pipelines, online experiments. We will not re-derive backpropagation. We will, however, spend a great deal of time on the parts of the system that the academic literature has historically under-emphasized: how training data is sourced, filtered, mixed, versioned, evaluated, and re-incorporated into itself. If you are an ML researcher who has built models but never operated a data pipeline at scale, you are the primary audience. If you are a data-platform engineer who has built pipelines but never had to defend a model's downstream metrics, you are also the primary audience. The thesis is that these two roles are converging, and that the most consequential engineering decisions in modern AI happen at the seam between them.

---

1.1 The Saturation of Pre-training Scale Laws

1.1 预训练规模法则的饱和

The argument for data as the binding constraint on AI capability rests on a quantitative claim about the shape of the loss landscape. We make that claim precise in this section, with the caveat that the underlying empirical literature is moving quickly and the specific numbers cited here are illustrative of the *shape* of the result rather than the exact functional form.

1.1.1 From Kaplan to Chinchilla: a brief history of scaling

The Kaplan scaling laws (Kaplan et al., 2020) established that, for autoregressive language models in the compute range of interest at the time, the test loss $L$ obeyed a smooth power law in the number of parameters $N$, the dataset size $D$, and the compute budget $C$:

$$L(N, D) \approx \left(\frac{N_c}{N}\right)^{\alpha_N} + \left(\frac{D_c}{D}\right)^{\alpha_D} + L_\infty,$$

where $\alpha_N \approx 0.076$, $\alpha_D \approx 0.095$, and $L_\infty$ is the irreducible loss of the underlying data distribution. The key qualitative finding was that the exponents were such that, at the frontier of compute, it was cheaper (in loss reduction) to scale parameters than to scale data — a conclusion that shaped a generation of model releases from GPT-3 through Gopher to PaLM.

The Chinchilla result (Hoffmann et al., 2022) revised this conclusion. By training a family of models with carefully balanced $N$ and $D$, Hoffmann et al. showed that the *compute-optimal* allocation for a given FLOP budget was approximately $N \propto C^{0.5}$ and $D \propto C^{0.5}$, with the consequence that frontier models in 2022 had been dramatically under-trained. Their flagship Chinchilla model, at 70B parameters trained on 1.4T tokens, matched the much larger Gopher (280B parameters, 380B tokens) on a range of downstream evaluations despite being roughly four times smaller. The implication for data was direct: at any given compute budget, there existed a model size at which additional data yielded the maximum loss reduction, and the industry was, on average, spending too much of its data budget on too few parameters.

1.1.2 The post-Chinchilla evidence: compute-optimal allocation and its limits

The next eighteen months produced a more uncomfortable follow-up. As labs pushed the Chinchilla envelope to 10x and 100x more compute, two empirical regularities began to emerge. First, the Chinchilla exponents themselves appeared to be shifting: at the largest scales, the optimal ratio of tokens to parameters was increasing, not constant, with some estimates suggesting $D/N$ rising toward 20 or 30 tokens per parameter for the most capable models. Second, the returns to adding *raw* web text were clearly sublinear in a way that the Kaplan functional form had not predicted. After a few hundred billion tokens of common-crawl-derived data, additional tokens drawn from the same distribution yielded progressively smaller improvements in pretraining loss, and even smaller improvements in the downstream behaviors that actually matter — reasoning, factuality, instruction-following, code generation.

A useful way to see this is to plot the loss-reduction-per-token as a function of cumulative tokens seen. Empirically, this curve has the shape of a saturating exponential or a stretched power law: the first 100B tokens of a high-quality corpus can drop the loss by more than the next 1T tokens, which can in turn drop it by more than the next 10T. Th