GPU 101: Understanding the Core Hardware That Powers Modern AI

By 2012, just as deep learning was beginning to move from academic curiosity to industrial application, NVIDIA’s CUDA ecosystem had matured enough to let researchers tap into that parallel power without needing to write low-level graphics code. That alignment of hardware capability, software accessibility, and market demand is the first key to understanding why NVIDIA, not some other chipmaker, came to dominate AI infrastructure. To make that connection clear, we first need to demystify the technical specifications that define GPU performance for AI workloads, and see exactly how those specifications map to the unique demands of modern AI systems.

Decoding GPU Core Specifications for AI Use Cases

When shopping for a GPU for AI work, you will encounter a long list of technical terms: CUDA cores, tensor cores, VRAM capacity, memory bandwidth, TDP, TFLOPS, and more. For a beginner, these can feel like alphabet soup, but each term describes a concrete feature that directly impacts how fast and how large of an AI model you can run. We will break each down using a simple analogy: think of a GPU as a factory for performing mathematical calculations, the core of which is building and training AI models.

CUDA Cores: The Factory Workers

CUDA cores are the basic arithmetic processing units of an NVIDIA GPU, analogous to individual workers on a factory floor. Each core can perform simple floating-point calculations (additions, multiplications) on small pieces of data at a time. Unlike CPU cores, which are designed to handle complex, sequential tasks (like running an operating system or a single-threaded application) with very low latency, CUDA cores are optimized for high-throughput, parallel execution of thousands of simple, identical tasks at once. To put the scale in perspective: a high-end consumer GPU like the NVIDIA GeForce RTX 4090 has 16,384 CUDA cores, while a top-tier consumer CPU like the Intel Core i9-14900K has only 24 total cores (8 performance cores + 16 efficiency cores). Even a data center CPU designed for parallel workloads, like the AMD EPYC 9654, has just 96 cores. That raw core count gap is the first reason GPUs outperform CPUs for AI: AI workloads are almost entirely made up of thousands or millions of small, independent mathematical operations that can be split across all those cores simultaneously. A common point of confusion is the difference between CUDA cores and the older term “stream processors” used by AMD. The two are functionally identical: both refer to the basic shader processing units of a GPU, just branded differently by their manufacturers. For the purposes of this book, we will use “CUDA cores” to refer to NVIDIA’s implementation, and “stream processors” for AMD’s, but the underlying parallel processing principle is the same for both.

Tensor Cores: The Specialized AI Workers

Starting with the Volta architecture in 2017, NVIDIA added a second type of processing unit to its data center GPUs: tensor cores. If CUDA cores are generalist factory workers who can do any simple calculation, tensor cores are specialized workers trained exclusively to perform the matrix multiplication operations that make up 90%+ of the computation in modern AI models. Matrix multiplication is the foundational operation of nearly all AI systems: when a large language model processes a prompt, it multiplies the input token embeddings by a series of weight matrices to produce output probabilities; when a computer vision model classifies an image, it multiplies pixel data by convolutional weight matrices to extract features. Tensor cores are hardwired to perform these matrix multiplications far faster than generalist CUDA cores, especially when using lower-precision number formats (we will explain precision in detail later in this chapter). The performance gap is staggering: the NVIDIA H100 GPU, NVIDIA’s current flagship data center chip, has a peak FP8 (8-bit floating point) tensor core performance of 1,979 teraFLOPS (trillion floating-point operations per second), compared to just 67 teraFLOPS of FP32 (32-bit floating point) performance from its generalist CUDA cores. That 30x speedup for AI-specific operations is why tensor cores are the single biggest factor driving AI performance gains in NVIDIA’s latest GPUs. For context, a 30x speedup means a task that took 30 days to run on a GPU without tensor cores can be completed in a single day on a modern H100.

> 💡 Digital Edition Sidebar: What Is a FLOP? > A FLOP (floating-point operation) is a single mathematical operation that works with decimal numbers, like multiplying two 0.5 values together. AI workloads are measured in FLOPS because they are almost entirely made up of these operations. A teraFLOP (TFLOPS) is 1 trillion FLOPs per second, a petaFLOP (PFLOPS) is 1,000 teraFLOPs, and an exaFLOP (EFLOPS) is 1,000 petaFLOPs. The world’s fastest supercomputer, Frontier, operates at 1.1 EFLOPS, roughly the combined FP64 performance of 16,000 H100 GPUs.

VRAM: The Factory Workbench

VRAM (Video Random Access Memory) is the on-board memory of a GPU, used to store the data being processed: model parameters, input data, intermediate calculation results, and output data. If CUDA cores are workers and tensor cores are specialized workers, VRAM is the workbench they use to hold their tools and materials. The larger the workbench, the more data you can have on hand without stopping work to fetch more from the system’s main RAM (which is much slower). For AI workloads, VRAM capacity is often the single biggest limiting factor for how large a model you can run. AI models are measured in parameters: a 7-billion-parameter model like Llama 3 7B has 7 billion individual weight values that need to be stored in VRAM to run. Each parameter stored in 16-bit floating point (FP16, the most common precision for AI training) takes up 2 bytes of memory, so a 7B model requires 14 GB of VRAM just for its weights, plus additional space for input data, intermediate results, and the operating system running on the GPU. That means a GPU with less than 16 GB of VRAM cannot run a 7B model at all, no matter how fast its cores are. As models grow larger, VRAM requirements scale linearly: a 70B model requires 140 GB of VRAM for FP16 weights, and a 1-trillion-parameter model requires 2 TB of VRAM. That is why the largest AI models require dozens or even hundreds of GPUs working together, each with 80 GB of high-speed VRAM, to split the model across multiple devices.

Memory Bandwidth: The Speed of the Factory Supply Line

VRAM capacity tells you how much data you can store on the GPU, but memory bandwidth tells you how fast you can move data between the VRAM and the processing cores. If VRAM is the workbench, memory bandwidth is the speed of the conveyor belt that moves tools and materials to the workers. If the conveyor belt is too slow, even the fastest workers will sit idle waiting for parts, drastically reducing overall performance. Memory bandwidth is measured in GB/s (gigabytes per second). High-end AI GPUs use a type of high-speed memory called HBM (High Bandwidth Memory) stacked directly on top of the GPU die, connected via a ultra-wide memory bus. The NVIDIA H100, for example, uses HBM3 memory with a 5,120-bit wide bus, delivering 3,000 GB/s (3 TB/s) of bandwidth. By comparison, a high-end consumer GPU like the RTX 4090 uses GDDR6X memory with a 384-bit bus, delivering 1,008 GB/s of bandwidth—roughly 1/3 the speed of the H100. For AI workloads, bandwidth is often just as important as raw compute power. Large language models, for example, require constantly streaming weight data from VRAM to the tensor cores during inference; if bandwidth is too low, the tensor cores will be idle 50% or more of the time, wasting their massive compute potential. This is why NVIDIA’s data center GPUs prioritize HBM memory with extremely high bandwidth, while consumer GPUs use slower but cheaper GDDR memory optimized for gami