Why Are GPUs Used for AI? Matrix Math and Memory Explained

In everyday life

Ask a chat AI a question, and it writes back an answer. Search for "dog" in your phone's photo app, and it finds photos with dogs in them. Behind features like these, an enormous amount of calculation is going on.

News stories often say that building an AI takes thousands or tens of thousands of GPUs. For example, Meta trained the largest version of its text AI "Llama 3" on about 16,000 GPUs (a 2024 paper). But GPUs were first made to draw pictures on screens, for games and more. Why is a part made for drawing pictures used for AI?

Inside AI's calculations: "multiply, then add"

Most AI today is built on a set of calculation steps called a "neural network." The name comes from nerve cells in the brain, but what it does is arithmetic. It multiplies each input number by its own fixed number, then adds up the results. This "multiply, then add" is repeated, layer upon layer.

Let's look at a small example. Say we turn test scores (History 80, Math 60, English 70) into a "STEM score" and an "arts score." For STEM, we multiply History by 0.2, Math by 0.5 and English by 0.3, then add. For arts, we multiply by 0.4, 0.2 and 0.4, then add.

A table of multipliers laid out in rows and columns like this is called a matrix. The calculation in the figure is called matrix multiplication. You don't need formulas: thinking "for each row of the table, multiply by the inputs and add" is enough.

In AI, the inputs are text or images turned into numbers. The multipliers are called "weights," and they set how much each input counts. Real AI uses tables thousands of rows and columns wide, stacked in dozens of layers. In a 2017 explainer, Google said the heaviest part of running AI is usually matrix multiplication. To keep things simple, we have left out the small step done after each multiply-and-add.

Training and inference: a huge amount of math

Before an AI can be used, it goes through "training." It is shown many examples, and its weights are adjusted bit by bit so its answers move closer to the right ones. People do not set the weights; training does.

Once trained, the AI uses those fixed weights to answer new inputs. This is called inference. A chat AI writing a reply and a phone finding a dog in a photo are both doing inference.

The amount of calculation is enormous. GPT-3, a text AI released in 2020, had 175 billion weights. AIs like this use almost every weight once for each word they produce (strictly, for each unit called a "token"). Each weight takes one multiply and one add, so each word needs about 350 billion calculations in total (a rough guide).

Training takes about 3 times as much per word, about 1 trillion calculations. On top of computing the answer, it must work out how to adjust the weights. This "about 3 times" is a rule of thumb used in work such as Kaplan and colleagues' study (2020). GPT-3 was trained on about 300 billion tokens of text. No person with a calculator could ever keep up; it takes very fast machines.

CPU vs. GPU: a few versatile cores vs. many simple units

As we saw in "What Is a CPU?", a CPU takes a program's instructions one at a time and carries them out. One complete set of circuits that runs instructions is called a "core." A PC or phone CPU usually has from a few to a little over 20 cores. Each core has many tricks for getting complex work with many branches done quickly.

A GPU, on the other hand, lines up thousands of simple calculation units. Each one is less versatile than a CPU core. Instead, they can all do the same calculation on different data at the same time. Much of the chip's area goes to the calculation units themselves, not to tricks for handling branches.

Matrix multiplication fits this design perfectly. In the example above, the STEM and arts scores could be calculated separately, without waiting for each other. The same holds for large tables: each answer can be handed to its own calculation unit. Because thousands of answers are worked out at the same time, a GPU finishes far faster than a CPU. "CPU vs. GPU" compares the two in a table.

GPUs started out drawing pictures

A picture on a screen is a collection of tiny dots. Each dot is called a "pixel." A full HD screen has 1920 × 1080, or about 2.07 million, pixels. Many games recalculate the color of every pixel about 60 times a second. Every pixel's color is worked out with the same steps, so lining up many calculation units to work at the same time is efficient.

In 1999, NVIDIA released a product called the GeForce 256 and named it a "GPU." After that, GPUs' calculation units were redesigned so they could be used for calculations other than graphics. In 2006, NVIDIA announced CUDA, a system for doing general-purpose computing on GPUs.

The turning point for AI came in 2012. A team at the University of Toronto trained an image-recognition AI called "AlexNet" on two off-the-shelf gaming GPUs. This AI won an image-recognition contest by a wide margin. When it listed its top 5 guesses, the right answer was missing about 15% of the time; for the runner-up, about 26%. After this, GPUs came into wide use in AI research. The design built for drawing pictures, "the same math done by many units at once," fit AI's matrix math just as it was.

The other wall: how fast numbers can be moved

No matter how fast the calculation units are, they can only wait if the numbers they need do not arrive. The numbers are kept in memory. The amount of data that can be moved out of memory per second is called "memory bandwidth." In AI, bandwidth matters as much as calculation speed.

Take a text AI. As we saw, it reads almost every weight from memory for each word it produces. Say GPT-3's 175 billion weights are each stored as 16 bits (a string of sixteen 0s and 1s). With 8 bits to a byte, that is 2 bytes each, or about 350 GB (gigabytes) in total. Moving this many numbers for every word often takes longer than the calculation itself.

So GPUs for AI use a special memory called HBM (high bandwidth memory). HBM is a stack of memory chips placed right next to the GPU. It has a very large number of data paths side by side, so it moves many numbers at once. HBM3, whose standard was set in 2022, has 1,024 paths per stack.

NVIDIA's H100 GPU, announced in 2022, can move about 3.35 TB per second. A TB (terabyte) is 1,000 GB. For comparison, let's work out two sticks of DDR5-4800, a memory type for PCs. From the standard's numbers (4,800 million transfers per second, 8 bytes each), one stick moves 38.4 GB per second, and two move 76.8 GB. The H100 can move more than 40 times that.

Even so, moving numbers cannot keep up with calculation. With 16-bit numbers in matrices, the H100 can do about 1,000 trillion calculations per second (counting multiplies and adds separately). Compared with what it can move, that is about 300 calculations for every byte that arrives. Gholami and colleagues (2024) looked at the past 20 years. The peak calculation speed of server hardware grew about 3 times every 2 years. Memory bandwidth grew only about 1.6 times every 2 years. This problem, where moving numbers falls behind calculating, is called the memory wall.

So when running AI, systems reuse numbers as much as they can once they are moved. Chat AI services, for example, process many people's questions together. That way, weights read once can serve calculations for many people.

Shorter numbers, more calculations

In AI calculations, the answer often barely changes even if the numbers are a little less precise. So a common trick is to cut the number of 0s and 1s (bits) used for each number. The fewer the bits, the coarser the numbers that can be written.

Common number formats have names. A 32-bit number with a decimal point, often used as the standard for calculation, is FP32. In AI, the 16-bit FP16 and BF16, half its size, and the 8-bit integer INT8 are widely used. "FP32 vs. FP16 vs. BF16 vs. INT8" compares them in a table.

Shorter numbers bring three gains. Less data to move means the same memory bandwidth carries more numbers. More weights fit in memory. And the multiplying circuits shrink, so more calculation units fit in the same area and power.

Today's GPUs have dedicated circuits that do matrix multiplication on short numbers in bulk. NVIDIA calls them Tensor Cores. In the H100's specs, ordinary 32-bit math runs at about 67 trillion calculations per second. With Tensor Cores on 16-bit matrices, that rises about 15 times, to about 1,000 trillion. But cutting precision too far makes the answers drift, so how short the numbers can go is checked for each use.

Not only GPUs: TPUs and NPUs

Some AI chips are built only for matrix multiplication. The best-known is Google's TPU. Google began using TPUs in its own data centers in 2015 and announced them in 2016. The first TPU has 256 × 256 = 65,536 circuits that multiply and add 8-bit integers, laid out in a grid like a game board. It does matrix multiplication by passing numbers from each circuit to its neighbor. This approach is called a systolic array.

Phones and PCs also contain circuits made just for AI. These are called NPUs (neural processing units). For example, in 2017 Apple put its first NPU into a chip called the A11 Bionic. This chip was used in the iPhone 8 and iPhone X. According to Apple, it can do up to 600 billion calculations per second, and it powers features such as unlocking the phone with your face. NPUs focus on inference and are built to run on little power, so they suit battery-powered devices.

A GPU is a general-purpose part that handles graphics, AI and scientific computing. Dedicated chips such as TPUs and NPUs deliver more performance per watt on the math they are built for. In exchange, they are good for fewer jobs. "CPU vs. GPU vs. NPU" compares the three in a table.

Common misconceptions

By the numbers

The H100 figures are NVIDIA's specs for the SXM form of the H100. The 16-bit value uses Tensor Cores, without a trick that skips the zeros in tables with many zeros. The spec sheet's figure of about 2,000 trillion is the value with this trick.

Let's roughly estimate the effect of moving speed. Reading 350 GB of weights at the H100's 3.35 TB per second takes about 0.1 seconds. Even if the math took no time, each word would wait about 0.1 seconds for reading. In reality, 350 GB does not fit in one GPU (80 GB). So the weights are spread over 5 or more GPUs and read in parallel. Many people's requests are also processed together to cut the wait per person.

Learn more

• Compare in a table: DDR vs. LPDDR vs. GDDR vs. HBM (the kinds of memory used with GPUs)
• Try the numbers: the memory bandwidth calculator (works out the bandwidth of HBM, DDR5 and more)
• Terms for the next step: arithmetic intensity and the roofline model (how calculation speed and moving speed relate), mixed precision and data parallelism (ways to speed up AI calculation)
• Read about memory: "Memory vs Storage: What's the Difference?"
• Start from the foundation: "What Is a Transistor?" (calculation units are collections of tiny switches)