Why Your GPU Is Sweating: The Insatiable Hunger of Modern AI

Why Your GPU Is Sweating: The Insatiable Hunger of Modern AI

There is a smell in modern data centers that didn't exist fifteen years ago: the warm, slightly acrid breath of a graphics card running flat out. Walk into any cloud region in 2025, and the racks hum with a sound that is half turbine, half sea of fans. A single H100, the workhorse of the moment, dissipates 700 watts—enough to toast a small Thanksgiving turkey, and it does so not because of bad engineering but because the math is simply relentless. Behind every chatbot response, every image generation, every "would you like me to expand on that," sits a silicon beast pulling multi-megawatt power budgets to keep up with a hunger that grows on a curve so steep it makes Moore's Law look like a flatline.

This book is about those beasts. But before we open the cage, we have to understand what they are feeding on. The 2020s are the decade in which a single class of software—the transformer neural network—consumed the entire trajectory of general-purpose computing and demanded something better. To see why this happened, we need to do three things in this opening chapter. First, we need to follow the numbers from AlexNet in 2012 to the trillion-parameter frontier of 2026 and watch the compute curve bend upward. Second, we need to do the arithmetic ourselves, because the relationship between parameters, tokens, and FLOPs is the single most useful equation in modern AI economics. Third, we need to understand why the CPU—the chip that ran the world for forty years—suddenly became the wrong tool, and what that wrongness has to do with physics rather than bad design.

By the end of this chapter, you should be able to estimate the training cost of a frontier model on the back of a napkin, explain why your laptop sounds like a hair dryer when you load Llama 3 locally, and articulate why "general-purpose" stopped being a compliment in the datacenter. That last bit is the seed of everything that follows: the rise of tensor cores, the death of the megahertz race, the birth of entire silicon families (NVIDIA's data-center empire, Google's vertically integrated TPU, Huawei's ascendant Ascend), and the multi-trillion-dollar infrastructure buildout that will define the rest of the decade.

1.1 The Compute Explosion: A Curve That Broke the Chart

To understand the AI silicon revolution, you have to understand the curve. Not the abstract, hand-wavy "AI is growing fast" curve that appears in pitch decks, but the actual, on-the-ground, kilowatt-hours-in-the-real-world curve. The story begins on a quiet evening in late September 2012, when a group of researchers from the University of Toronto submitted a paper to a computer vision conference called NeurIPS. The paper was called "ImageNet Classification with Deep Convolutional Neural Networks." The system was called AlexNet. Its authors, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, trained it on two NVIDIA GTX 580 GPUs—a card that, by modern standards, was a toy. The training run consumed roughly $1.4 \times 10^8$ FLOPs per image, the model had 60 million parameters, and the entire thing took about five to six days on a pair of consumer cards you could buy at Best Buy. The result was not just a paper; it was a detonation. AlexNet cut the error rate on ImageNet in half, and it did so using a method that would not get better by cleverer algorithms. It would get better by being bigger.

The next eleven years produced a curve so steep that anyone who plotted it on linear graph paper ran out of paper. In 2014, VGG and Inception pushed the parameter count into the hundreds of millions. In 2015, ResNet, the architecture that introduced skip connections and made very deep networks practical, took us to 25 million parameters and beyond, while training compute per run climbed past $10^{18}$ FLOPs. By 2017, the Transformer architecture—originally proposed for machine translation—was applied to language modeling, and the curve went vertical. BERT in 2018 was trained on $7 \times 10^{10}$ FLOPs. GPT-2 in 2019 was $1.5 \times 10^{21}$. GPT-3 in 2020 was roughly $3.14 \times 10^{23}$ FLOPs—more than three orders of magnitude larger than BERT and equivalent to running a single high-end CPU core continuously for about a million years. And then, in the next three years, we did it again, and again, and again.

The following diagram captures the trajectory. Each node is a landmark model with its approximate training compute. The Y-axis is logarithmic; on a linear scale, every model after 2018 would be invisible against the wall that GPT-3 and its successors have become.

```mermaid flowchart LR A["2012 AlexNet<br/>~1.4e8 FLOPs<br/>60M params"] --> B["2015 ResNet-152<br/>~4e10 FLOPs<br/>60M params"] B --> C["2018 BERT-Large<br/>~7e10 FLOPs<br/>340M params"] C --> D["2020 GPT-3 175B<br/>~3.14e23 FLOPs<br/>175B params"] D --> E["2022 PaLM 540B<br/>~2.5e24 FLOPs<br/>540B params"] E --> F["2023 GPT-4<br/>~2e25 FLOPs<br/>~1.8T params (est.)"] F --> G["2024 Llama 3.1 405B<br/>~4e25 FLOPs<br/>405B params"] G --> H["2025-26 Frontier<br/>1e26 to 1e27 FLOPs<br/>Multi-Trillion params"] ```

What you are looking at is not a normal technology adoption curve. It is closer to the curve of a chain reaction, the kind of thing you see in epidemiology or in finance just before someone at the central bank picks up the phone. From 2012 to 2020, the compute used to train the best published model doubled approximately every six months. That is *four times* faster than the doubling time of Moore's Law, which is itself a doubling every two years. After 2020, the doubling slowed only slightly—frontier labs hit practical walls of capital, electricity, and chip availability, not theoretical limits—and even with that slow-down, the world still trains models today that would have been considered science fiction in 2018.

The phrase that captured this moment in 2018, in a paper that has since become one of the most-cited works in computer science, was "Compute and Parameters as Predictors of Model Loss." Kaplan and colleagues at OpenAI noticed something almost embarrassing in its simplicity: if you train a transformer language model with $N$ parameters on $D$ tokens, the cross-entropy loss scales as a smooth power law in both $N$ and $D$. They fitted the curves, and the result was clean. Bigger is better, predictably better, and the better-ness is smooth. There are no cliffs, no magic thresholds where 100 billion parameters suddenly unlock sentience. Just a power law. A year later, DeepMind's "Chinchilla" paper refined the picture and gave us a punchline: for a given compute budget $C$, the optimal model is approximately the one that uses $N$ parameters trained on $D$ tokens such that $C \approx 6ND$. In English: to get the most out of your flops, you should scale parameters and data roughly together. If you have the flops, you should grow both. The implication was crushing for any organization that had built its strategy around a 1.5-billion-parameter model on a billion tokens. The leaderboard was about to skip several leagues.

What the scaling laws revealed, and what the next five years confirmed, is that the AI compute curve is not a temporary anomaly. It is the underlying physical reality of a class of algorithms whose performance is monotonic in the resources you pour into them. This is almost unique in the history of computing. Most software saturates. Most algorithms hit a point of diminishing returns. But transformers keep getting better, smoothly, in a way that makes "scaling is all you need" both a famous meme and a literal business plan. The cost of that business plan is, however, the central problem of this chapter: training a frontier model in 2025 takes about 50 to 100 megawatt-days of compute, a sum of money that crosses the billion-dollar threshold for the largest runs, and an energy footprint that is now measured as a percentage of national electricity grids in some countries. The chi