CPU vs GPU: Differences in Architecture and Use

A CPU is built to finish one task as fast as possible, so it has a few powerful cores with large caches and smart control logic. A GPU is built to finish many similar tasks at once, so it has thousands of simple arithmetic units and hides memory delays by switching between huge numbers of threads. CPUs are good at complex, branching, serial work; GPUs are good at massively parallel math such as graphics and deep learning.

AspectCPUGPU
Design goalLow latency: finish a single thread as fast as possibleHigh throughput: finish as much total work as possible
Core countA few to a few hundred complex coresThousands of simple arithmetic units, grouped into larger blocks
Execution modelOut-of-order execution and branch prediction in each core, plus SIMD instructionsSIMT: groups of threads (for example 32) run the same instruction together
Control logic and cacheLarge caches and sophisticated control logic take much of the chip areaMore area goes to arithmetic units; caches are smaller per thread
Hiding memory latencyMostly with caches and prefetchingMostly by switching to other ready threads while one waits
MemoryDDR or LPDDR, typically tens to a few hundred GB/s of bandwidthGDDR or HBM, typically hundreds of GB/s to several TB/s of bandwidth
BranchingHandles branches and irregular code wellSlows down when threads in a group take different branches (divergence)
Data locationWorks directly on main memoryA discrete GPU has its own memory, so data must be copied over a bus such as PCIe
ProgrammingAlmost any language and toolSpecialized models such as CUDA, HIP, OpenCL and graphics shader languages
Typical workloadsOperating systems, databases, control logic, general applicationsGraphics, matrix and vector math, deep learning, simulation

Figures are typical orders of magnitude and vary by process generation and product.

When to use which

CPU
When the work is serial, branchy or needs fast response from one thread: running the operating system, application logic, databases.
GPU
When the same operation is applied to a huge amount of data: rendering images, training and running neural networks, scientific simulation.
Together
Most systems use both. The CPU runs the program and prepares the data, and the GPU does the heavy parallel math. Sending too little work to a GPU can be slower than staying on the CPU, because of the cost of copying data.

FAQ

Why is a GPU faster than a CPU for deep learning?

Deep learning is mostly matrix math that can be split into many identical operations. A GPU has thousands of simple arithmetic units and very high memory bandwidth, so it handles this parallel work far better than the few complex cores of a CPU.

Can a GPU replace a CPU?

No. GPUs are poor at branchy, serial code and need a CPU to run the operating system and feed them data. The two work together.

What does SIMT mean on a GPU?

SIMT (single instruction, multiple threads) means a group of threads, for example 32, executes the same instruction together on different data. If threads in a group take different branches, they run one after another, which is called divergence.

Why can using a GPU be slower for small tasks?

Data must be copied between CPU memory and GPU memory over a bus such as PCIe. For a small amount of work, the copy time can be larger than the time saved by the GPU.

Check your understanding

How does the GPU SIMT (single instruction, multiple threads) model differ from CPU SIMD instructions?

  1. ASIMT applies many instructions to one datum; SIMD applies one instruction to one datum
  2. BSIMT and SIMD are exactly the same with different names
  3. CIn SIMT each thread can be written as an independent program and the hardware handles branches; with SIMD the programmer or compiler explicitly uses fixed-width vector instructions
Show answer and explanation

Answer: C. In SIMT each thread can be written as an independent program and the hardware handles branches; with SIMD the programmer or compiler explicitly uses fixed-width vector instructions

CPU SIMD applies one instruction to a vector register holding 4 or 8 data items, so programs are written with the vector width in mind. With GPU SIMT the programmer writes ordinary code for a single thread, and the hardware bundles 32 threads to execute in lockstep. Threads may branch differently; the hardware executes the divergent paths one after another, though efficiency drops when it does.

What is the main reason GPUs are well suited to matrix operations and AI training and inference?

  1. ATheir clock frequency is over 100 times that of CPUs
  2. BThey have many simple arithmetic units and can process large amounts of data in parallel
  3. CThey have no memory
Show answer and explanation

Answer: B. They have many simple arithmetic units and can process large amounts of data in parallel

GPUs were built to process many screen pixels at once, packing thousands to over ten thousand simple arithmetic units. Workloads like matrix math that repeat the same computation over huge data sets exploit this parallelism for high performance. Modern GPUs add dedicated matrix units (such as Tensor Cores) and high-bandwidth HBM, making them the workhorse of AI training. CPUs remain better at branch-heavy sequential code.

In general, how do GPU caches compare with CPU caches?

  1. AGPUs have no caches at all
  2. BGPUs have L1 and L2 caches, but with so many concurrent threads the cache capacity per thread is far smaller
  3. CGPU caches are many times larger per thread than CPU caches
Show answer and explanation

Answer: B. GPUs have L1 and L2 caches, but with so many concurrent threads the cache capacity per thread is far smaller

GPUs have an L1 cache per SM and an L2 cache shared across the chip, but with tens of thousands of threads running simultaneously, the capacity available per thread is orders of magnitude smaller than for a CPU thread. Rather than relying on caches to reduce latency, GPUs hide latency by switching among many warps and depend on coalescing and shared memory to use memory well.

  1. In a program using a discrete GPU, what is generally true about CPU-side and GPU-side memory?
  2. Sending data to a GPU for light work such as adding array elements once can be slower than computing on the CPU. What is the main reason?
  3. Why does pinning (page-locking) CPU memory before transferring it to the GPU speed up the transfer?

Practice GPU Programming questions

Key terms explained

GPU
Short for graphics processing unit: a processor with many simple execution units that processes large amounts of data in parallel. Besides graphics, it is widely used for AI training and inference.
SIMD
Short for single instruction, multiple data: one instruction performs the same operation on several data elements at once. It speeds up image processing and AI calculations.
Cache memory
Fast, small memory placed close to the CPU. It holds frequently used data to bridge the speed gap with main memory. Mostly built from SRAM.
HBM
High Bandwidth Memory. Multiple DRAM dies stacked with TSVs, achieving very high bandwidth through an extremely wide bus.

Related comparisons