Why Data Is the New Model (and Why You're Probably Underinvesting)

Your model is not the bottleneck. The model is the easiest thing to upgrade. Swap in a bigger one, fine-tune for a weekend, rent some H100s, ship it. Done. The thing that's actually blocking you—the thing that quietly costs you six months of lost revenue and a measurable chunk of your engineering team's sanity—is data. Specifically, it's the absence of a working *data engine*: a system that turns raw signal into high-quality training signal at the speed your roadmap demands. And the worst part is that almost nobody admits this out loud, because admitting it means admitting that the last two years of model-centric strategy were, at best, premature.

Let's start with a number. In 2022, DeepMind published Chinchilla, and the field of compute-optimal scaling was born. The headline finding was simple enough to fit on a slide: for a given compute budget $C$, the optimal model size $N$ and dataset size $D$ should grow together, roughly along $D \approx 20N$. Train a smaller model on much more data, and you almost always beat a bigger model on less data, for the same compute. The follow-up papers (Llama 2, Llama 3, the various "data-optimal" studies) kept sharpening the knife: beyond a certain point, additional parameters are mostly a rounding error, and additional high-quality tokens are where the actual gains live.

But here's the part the slide doesn't say. Most enterprise teams are not training foundation models. Most enterprise teams are fine-tuning or post-training a base model on a domain-specific corpus that is, charitably, a mess. And the relevant optimization frontier for those teams is not the Chinchilla frontier—it's the "we have a 12-month roadmap and need to hit 95% accuracy on the 200 task types our customers actually use" frontier. On that frontier, the dominant lever is data quality, not model size. We can prove it with back-of-the-envelope math, and we will, but first we want to name the disease: underinvestment in data is the single most common cause of failed AI initiatives, and almost every team has a story about it.

So this chapter is a kind of intervention. We're going to walk through the empirical case for why data is the new model, look at the actual cost curves that determine where you should spend your next dollar, dissect what a modern data team actually looks like, explain why the naive "just get more data" advice is a trap, and introduce the three-pillar framework—quality, coverage, feedback—that will run through the rest of the book. By the end, you should be able to score your own organization's data maturity and walk into a planning meeting with a one-page charter that will, if you defend it well, change what your team works on for the next four quarters.

The Empirical Case: Why Data Beats Parameters in Most Settings

Let's get the math out of the way early, because the math is what makes the rest of this chapter feel less like opinion and more like an engineering argument.

A standard transformer language model trained with cross-entropy loss follows what's called a *neural scaling law*. Empirically, the loss $L$ as a function of model size $N$, dataset size $D$, and compute $C$ can be fit with power laws of the form:

$$L(N, D) \approx \left( \frac{N_c}{N} \right)^{\alpha_N} + \left( \frac{D_c}{D} \right)^{\alpha_D} + L_\infty$$

where $N_c$ and $D_c$ are constants, $\alpha_N$ and $\alpha_D$ are exponents typically around 0.05 to 0.1, and $L_\infty$ is the irreducible loss floor for a given distribution. The first term says: bigger models reduce loss. The second term says: more data reduces loss. The third term says: there is a floor you cannot break no matter what you do.

Two things follow that are deeply inconvenient for the model-maximalist worldview. First, both exponents are small, which means neither lever gives you a linear return; every doubling of $N$ or $D$ buys you a smaller and smaller drop in loss. Second—and this is the one that matters for us—the exponent on $D$ is at least as large as the exponent on $N$, and in well-tuned, well-curated training runs, it's larger. Empirically, when you improve data quality (a notion we'll define precisely later), you're effectively shifting the second term down by increasing $D_c$, and that shift compounds.

Now let me translate that into the language of an actual product. Suppose you're building a customer support copilot. Your current fine-tuned model sits at 78% accuracy on your eval set. You have a choice: spend the next quarter fine-tuning a bigger base model, or spend the next quarter curating 5x the high-quality support tickets. Both are valid engineering projects. Which one wins?

The Chinchilla math says: probably the data. The reason is that your eval set is, almost certainly, dominated by data that is qualitatively different from what your base model trained on. The base model has seen billions of tokens of generic web text. It has not seen your tickets. The marginal value of the 5 millionth pre-training token is tiny. The marginal value of the 50,000th *curated* support ticket, paired with a high-quality human-labeled resolution, is enormous. The data is the bottleneck because the data is the only place where domain signal actually lives.

This isn't just a theoretical argument. We can see it in three public case studies that, taken together, will convince you—if the math didn't already.

Case Study #1: Google's RETVec and the Spam Filter

In 2023, Google's spam and abuse team published a paper on RETVec, a multilingual, robust text encoder they built to replace the legacy spam classifier in Gmail. The headline result was striking: 38% relative improvement in spam detection at the same false-positive rate, and a 19% reduction in false positives at the same recall. The architecture was interesting but not magical—a character-level encoder with an embedding table designed to be robust to adversarial perturbations. What was actually magical was the data.

Specifically, the team built a synthetic data generation pipeline that produced adversarial variants of real spam emails: homoglyphs, zero-width characters, inserted punctuation, transliterations across scripts, base64-encoded payloads, and so on. The training set was the legacy model's *false negatives*—the spam it had failed to catch—augmented with these synthetic perturbations and relabeled by a combination of automated checks and human review.

The lesson here is not "use character-level encoders." The lesson is that a model trained on the right slice of the data distribution beat a bigger model trained on a generic slice. The team's most senior engineer said, in a podcast appearance later, that the architectural change was roughly 20% of the win, and the data work was the other 80%. That's the data flywheel in miniature: find where the model fails, generate better data for exactly that region, retrain, redeploy, repeat.

Case Study #2: Tesla's Occupancy Networks

In 2022, Tesla announced a major architectural shift in its Full Self-Driving stack: away from bounding-box detectors for cars, pedestrians, and lanes, and toward a single end-to-end occupancy network. The occupancy network takes a bird's-eye view representation of the world and outputs a voxelized grid—essentially, for every cell in 3D space around the car, what is the probability that this cell is occupied by something solid, and what is its semantic class?

This was a major architectural change, and it required a major data change. The training data was video from Tesla's fleet, but the *labels* were the hard part. To label a single clip for occupancy training, the team had to do multi-camera 3D reconstruction, motion-compensate the cameras, project human annotations into the voxel grid, and aggregate across many similar clips to get a clean signal. The result was a labeled dataset on the order of hundreds of millions of clips, each with billions of voxel labels.

Tesla's data team—which is now larger than its modeling team, by most