The AI Hardware Moment: Why Silicon Suddenly Matters Again

If you've spent any time in tech over the last fifteen years, you've probably heard some version of this sentence: "Software is eating the world." It was Marc Andreessen's famous 2011 provocation, and for a long time it looked prophetic. Apps ate taxis. Apps ate hotels. Apps ate retail, banking, media, and slowly, then suddenly, a great deal of everything else. The story was always a software story: clever algorithms, clever business models, clever distributions of *bits* that rearranged the value out of bulky *atoms*.

That story isn't wrong. It's just no longer the whole story.

Somewhere between the publication of "Attention Is All You Need" in 2017 and the release of GPT-4 in 2023, the locus of progress shifted. New software wasn't arriving *despite* hardware constraints — it was being defined by hardware. The transformer didn't just win because it was a good idea. It won because, on the silicon available at the time, it scaled in ways nothing else could. The result is something almost embarrassing for a discipline that had spent four decades pretending to be hardware-agnostic: you cannot understand modern AI without understanding the chips that run it. And you cannot understand the chips without understanding the three very different companies that bet, very early and very hard, on the idea that silicon would matter again.

This chapter is the overture. We're going to answer a deceptively simple question: *Why did AI workloads break the CPU model and force a new era of accelerator-first computing?* To answer it, we'll look at the workload itself — the math that turned out to be the killer app — and then at the slow-motion collapse of the assumptions that kept general-purpose CPUs at the center of the universe. We'll meet our three protagonists — NVIDIA, Google, and Huawei — each of whom saw the same future but bet on radically different ways to get there. We'll introduce a framework — *silicon, system, software* — that we'll use to compare their bets throughout the book. And we'll close with a practical mental model you can use to evaluate any AI accelerator you meet in the wild, from a phone NPU to a 2-megawatt training cluster.

Let's begin with the workload that broke everything.

The Workload That Broke the CPU

For nearly two decades, the dominant AI workload was a convolutional neural network. CNNs, as they were known, were charming: relatively small, friendly to caching, almost custom-built for the strengths of the hardware they ran on. A modern CPU could move between convolution, pooling, and activation functions with graceful jumps through memory hierarchy. A GPU could eat a CNN for breakfast because convolutions, like most of the math that mattered in image and speech workloads, decompose into matrix multiplies — exactly what graphics hardware had been optimized for since the late 1990s.

Then transformers showed up, and the rules of the game changed.

It Started with Attention

If you've read any serious AI writing in the last five years, you know the headline: transformers replaced recurrence and convolution as the default architecture for almost everything — language first, then vision, then speech, then biology, then code, then reasoning, then video. The mechanism that did the replacing is *attention*, expressed mathematically as:

$$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

Three matrices — Queries, Keys, and Values — get multiplied together. So do three pairs of learned projection matrices, one for each of Q, K, and V, before the attention block, and one big one after. Inside each transformer block, the operations are dominated by:

• A **batch** of large dense matrix multiplies for the Q/K/V projections (typically $\mathbb{R}^{n \times d_{\text{model}}}$, with $d_{\text{model}}$ in the thousands).
• A **$QK^\top$** multiplication producing an $n \times n$ attention matrix per head.
• A **softmax** (cheap, but memory-bandwidth hungry and tricky to do with low precision).
• A final **projection** back to model dimension.

The shape of this workload matters, so let me give you an intuition you can carry for the rest of the book. Imagine a sequence of 8,192 tokens, each represented by a 12,288-dimensional vector, processed across 96 attention heads. The Q projection, by itself, is roughly $n \times d_{\text{model}} \times d_{\text{model}}$ floating-point operations — about 10^15 operations per layer, per token batch, in FP32. In a 96-layer model like GPT-3's, you stack 96 of these on top of each other, and you begin to see why training a frontier model costs tens of millions of dollars in compute.

Now the key bit: almost all of these floating-point operations are dense matrix multiplies. Not sparse. Not structured. Dense. They sit on top of enormous matrices that don't fit in cache. They are bandwidth-bound at every step except the very heart of the matmul. And they are *embarrassingly parallel* — you can slice a matrix into thousands of tiles and hand each tile to a different arithmetic unit without anyone complaining.

That last property — embarrassingly parallel dense math with terrible locality — is the architectural fingerprint of the workload that ended the CPU era.

Why CPUs Couldn't Keep Up

A modern server CPU, even an excellent one like AMD's EPYC 9755 or Intel's Xeon 6, is a magnificent piece of engineering. It packs 128 to 192 cores onto a single die, runs at 3 to 4 GHz, and executes a small universe of instructions per cycle. It was designed, however, for a workload mix dominated by control flow, irregular memory access, and modest data parallelism — exactly the workload profile of an enterprise database, a web server, or an operating system scheduler.

Now compare that to a transformer block:

| Workload characteristic | Modern server CPU | Modern AI accelerator | |---|---|---| | Data type | FP32 / FP64 / INT64 general | FP16, BF16, INT8, FP4, FP6, FP8 | | Matrix size | Small, fits in cache | Enormous, exceeds cache by orders of magnitude | | Available parallelism | 16–192 cores per socket, deeply out-of-order cores inside | Thousands of small, lane-parallel MAC units | | Memory access pattern | Pointer-chasing, branchy | Sequential tile streams | | Required throughput | ~10s of GFLOPS/socket | ~1–2 PFLOPS/socket | | Power budget per FLOP | High | Low (after specialization) |

The mismatch is brutal. A CPU at 4 GHz issuing two FMAs per cycle across 128 cores delivers about $4 \times 10^9 \times 2 \times 128 \approx 1$ TFLOPS — about one-thousandth of what a single H100 delivers on BF16. To match a single accelerator's throughput with CPUs, you'd need a rack of them, plus a high-bandwidth fabric to keep them fed with bytes — and even then, the silicon area, the energy per useful operation, and the latency of the interconnect would all be worse by roughly an order of magnitude.

This is not an indictment of CPUs. CPUs are still the best thing ever built for general-purpose computing. The point is narrower: the workload that defines the next decade of computing is a workload CPUs were never designed for, and no amount of CPU cleverness was going to make them competitive at it.

The deeper reason is arithmetic intensity.

The Arithmetic Intensity Wall

Every time you do a multiply-accumulate, you need to fetch operands, do the math, and ideally write the result back to memory. The ratio of useful mathematical operations to bytes moved is called arithmetic intensity, usually measured in FLOPs per byte:

$$I_{\text{math}} = \frac{\text{FLOPs performed}}{\text{Bytes read + written}}$$

A workload is *compute-bound* when $I_{\text{math}}$ exceeds the ratio at which the hardware can move data from memory. It's *memory-bound* when it doesn't.

The killer fact about modern AI is that the workload has *so much* dense matrix math that the model itself — the weights — is the most-fetched thing in the universe. And the model is *enormous*. GPT