The Frontier Is Data

The Frontier Is Data

Where should the next engineering dollar go — into the model, or into the data? This is no longer an academic question. The frontier laboratories that built the strongest 2024–2025 generation of large language models — Meta with Llama 3, DeepSeek with V3, Google with the Gemma lineage, Mistral, and the closed labs whose details leak in waves of technical reports — have all answered it the same way. They did not allocate the marginal dollar to a cleverer architecture. They allocated it to the data engine. The evidence for that reallocation is empirical, not aesthetic, and it is now strong enough that any team still training on raw web crawls is paying a capability tax they can no longer afford. The argument of this book is straightforward: the binding constraint on modern AI capability is data, and the people who control the data engine control the frontier.

This opening chapter establishes the empirical case and the working vocabulary the rest of the book relies on. We work through four moves. First, we examine the isoFLOP evidence that has shifted the field's intuition since the original Chinchilla paper, and show why data quality now dominates data quantity on the steep part of the capability curve. Second, we read the public technical reports from Llama 3, DeepSeek-V3, and the Gemma lineage to extract what frontier labs are actually doing — the same conclusion, reached independently, from three very different provenance paths. Third, we install three working definitions (data quality, data engine, data flywheel) that we will use without restating throughout the remaining chapters. Fourth, we provide a diagnostic framework and a worked reconstruction of a frontier pipeline so the reader can locate their own organization on the same map. The chapter closes with an explicit recommendation about how to think about the next engineering dollar.

1.1 The Post-Chinchilla Evidence: Why Quality Beats Quantity Beyond the Compute Knee

Hoffmann and colleagues' 2022 Chinchilla result established the rule of thumb that, for a fixed compute budget, training compute is best spent roughly equally across model parameters and training tokens. The rule was enormously influential because it gave the field a knob to turn: at any given FLOP budget, the loss-minimizing choice was a specific parameter count and a specific token count, both of which scaled together. Practitioners read Chinchilla as a prescription to *train on more data*. That reading is half right and half wrong, and the wrong half has been quietly corrected over the last two years by a sequence of careful isoFLOP studies that have changed what "more data" means.

The correction has two parts. The first part is that, beyond a certain compute regime, simply adding more tokens — even of decent quality — yields diminishing returns that are much steeper than the original Chinchilla fit predicted. The second part is that, in the same regime, *quality-thresholded* data continues to deliver large capability gains. Together, these two findings redefine the meaning of "compute-optimal training": in the regime where frontier models actually live, the binding factor is not how many tokens you can buy, but how many tokens of sufficient quality you can produce.

The empirical basis for this redefinition is a sequence of isoFLOP studies that vary the data mix while holding total tokens fixed. The cleanest version, and the one most often cited, is the body of work associated with the Llama 3 technical report and with concurrent independent analyses. The methodology is the same. You fix a compute budget $C$. You sweep over several candidate mixes, where each mix is described by the number of tokens $N$ drawn from each of several quality tiers — call them $T_1$ (highest quality, including curated human text, distilled model output, and reference-grade code), $T_2$ (filtered web text), and $T_3$ (unfiltered web text). For each mix, you train to completion at compute $C$ and measure a battery of capability benchmarks. The isoFLOP surface is the resulting map from mix to capability.

The empirical shape of that surface is what changed. In the small-compute regime — say, training a 1B parameter model on a few hundred billion tokens — adding $T_3$ tokens behaves roughly the way Chinchilla suggested: capability rises with the log of tokens, and the marginal benefit of upgrading a token from $T_3$ to $T_2$ is small. In this regime, the loss is dominated by the model's inability to absorb the data, not by the data's intrinsic quality. Below the compute knee, you should be data-hungry.

Above the knee — and the knee in 2025 sits at somewhere between $10^{22}$ and $10^{23}$ training FLOPs for the architectures that work in practice — the surface rotates. A 70B-class model trained on 5T tokens of unfiltered web text is dominated by a 70B-class model trained on 2T tokens of carefully curated data across most capability benchmarks. The reason is that, in this regime, the model is no longer the bottleneck. The model can absorb what it is given. The question is what it is given. The capability surface tilts so that the gradient points along the *quality* axis, not the *quantity* axis.

We can express this more formally. Let $L(C, N, q)$ be the achievable loss at compute $C$, token count $N$, and an effective data-quality parameter $q \in [0, 1]$ that summarizes the per-token quality of the mix. Chinchilla treats $L$ as roughly independent of $q$ for fixed $C$ and $N$, which is the source of the prescription to scale $N$ with $C$. The empirical isoFLOP studies in the post-Chinchilla regime find a different functional form:

$$\frac{\partial L}{\partial q}\bigg|_{C, N} \gg \frac{1}{N} \cdot \frac{\partial L}{\partial \log N}\bigg|_{C, q}$$

In words: a unit improvement in data quality buys more loss reduction than the same unit of compute spent on more tokens. The exact ratio depends on the regime and the capability being measured, but the qualitative finding has been replicated across multiple labs. The PyTorch team's analyses, the BigScience effort, the Llama 3 paper, and various independent labs have all reported some version of this.

A second body of evidence reinforces the same conclusion from a different direction. When labs train past the original Chinchilla token count — so-called "over-training" — they do so by adding high-quality tokens, not by repeating or padding with low-quality ones. DeepSeek-V3 trained on 14.8T tokens; Llama 3 trained on roughly 15.6T tokens; both models are at the upper end of what was considered compute-optimal for their parameter counts under the original Chinchilla rule. Neither report credits raw crawl scale for the capability gains. Both credit filtering, curation, and — particularly in the case of code and reasoning data — synthetic generation from earlier-generation models.

The practical implication is sharp. The marginal token of unfiltered Common Crawl is, in the modern regime, almost worthless. The marginal token of high-quality human or distilled data is the dominant lever. Teams that have not yet built a data engine capable of producing that token are paying a tax that compounds with every training run.

A word of caution, because the field has been here before. The "data quality matters" claim is not new. What is new is the *magnitude* of the effect, and the clarity with which isoFLOP studies have measured it. The risk of an overcorrection is real. There is a temptation, especially in engineering cultures that love clean stories, to read the isoFLOP evidence as "quality is all that matters" and to abandon quantity entirely. The corrected reading is the one we will use throughout the book: *in the regime where frontier models live, capability is most sensitive to the highest-quality tokens in the mix, but the mix still requires a long tail of lower-quality tokens to cover distributional ground that high-quality sources do not span*. The frontier labs have not aban