English
Menu

CPU vs GPU vs NPU: Key Differences Explained

A CPU is a general-purpose processor that can run anything, using a small number of complex cores to finish a single task quickly. A GPU processes large amounts of data in parallel with many compute units and is used for a wide range of parallel computing, including graphics and AI training. An NPU is dedicated circuitry specialized for the matrix multiply-accumulate operations that make up most of the computation in neural networks; by arraying many low-precision units such as INT8 and reusing data in on-chip memory, it performs inference with very high power efficiency. In return, it handles poorly any operations it does not support.

AspectCPUGPUNPU
Design philosophyGeneral-purpose, low latency. Finish a single task as fast as possibleHigh throughput. Handle large amounts of parallel computation togetherSpecialized for neural network operations (mainly inference), maximizing performance per watt
Compute unit organizationA few to a few hundred complex cores, each with SIMD unitsThousands of simple compute units; many also have matrix unitsA large array of multiply-accumulate units (such as a systolic array), plus vector units for activation functions and the like
ControlBranch prediction and speculative (out-of-order) execution per instructionSwitches among many threads to hide memory latencyMost follow a data flow determined in advance by the compiler, with little per-instruction control
StrengthsBranch-heavy code, operating systems, application logicGraphics, AI training and inference, scientific computingAI inference centered on convolutions and matrix multiplication
MemoryLarge caches and main memory such as DDRHigh-bandwidth memory such as GDDR and HBMReuses weights and intermediate data in on-chip SRAM buffers. When built into an SoC, shares main memory with the CPU
Main data typesAlmost anything, centered on FP64, FP32 and integersMainly FP32, FP16 and BF16. Also FP8 and others for trainingMainly low precision such as INT8, INT4 and FP16
Performance metricsClock frequency, core count, IPCFLOPS (TFLOPS, etc.)TOPS (usually a theoretical peak counted in INT8)
Power efficiency for AI inferenceLowHighHighest. Can be an order of magnitude or more higher than a CPU
FlexibilityHighest. Runs any programHigh. Broadly usable for parallelizable workloadsLow. Unsupported operations fall back to the CPU or GPU
ProgrammingGeneral-purpose programming languages and compilersKernels written in a GPU parallel programming modelTrained models are converted and run by AI compilers and inference runtimes. Rarely programmed directly
Where it is foundEvery kind of computerDiscrete cards, or built into SoCsOften built into smartphone and PC SoCs. Dedicated chips for data centers also exist

Figures are typical orders of magnitude and vary by process generation and product.

When to use which

CPU
Branch-heavy, sequential processing and control of the whole system. Also used to run small models and operations the NPU does not support.
GPU
Large-scale parallel computation that needs a choice of precisions and flexibility: AI training, inference of large models, graphics, scientific computing, etc.
NPU
When you want inference such as photo enhancement, speech recognition and translation to run constantly on smartphones and PCs without draining the battery. Models are quantized and run in formats the NPU handles well, such as INT8.

FAQ

What is the difference between an NPU and a GPU?

A GPU is a general-purpose parallel processor whose many compute units can be used flexibly by programs, and it can also handle graphics and training. An NPU focuses on the multiply-accumulate operations of neural network inference and is built around low-precision compute units and on-chip memory, so it is highly power-efficient for the workloads it supports but handles other workloads poorly.

Why is an NPU so power-efficient?

A CPU activates many circuits for each instruction, such as branch prediction and instruction reordering, whereas an NPU omits most of them and arrays many small multiply-accumulate units for formats such as INT8. It also reuses weights and intermediate data in on-chip SRAM, reducing energy-hungry accesses to external memory.

Can an NPU be used for training?

NPUs in smartphones and PCs are designed mainly for inference. Training requires gradient computation and higher precision, so GPUs or training accelerators for data centers are generally used.

Is an NPU with a higher TOPS number always faster?

Not necessarily. TOPS is usually a theoretical peak counted in INT8 or similar, and sometimes assumes sparse data. Actual speed varies greatly with memory bandwidth, compute utilization, and whether the model's operations are supported by the NPU.

Check your understanding

Why are the NPUs in smartphone SoCs more power-efficient than CPUs for AI inference?

  1. ANPUs compute with light instead of electricity
  2. BThey pack many low-precision (such as INT8) multiply-accumulate units and omit general-purpose circuitry unneeded for AI
  3. CNPUs always run at higher clock speeds than CPUs
Show answer and explanation

Answer: B. They pack many low-precision (such as INT8) multiply-accumulate units and omit general-purpose circuitry unneeded for AI

To run anything, a CPU exercises much circuitry per instruction, such as branch prediction and instruction reordering. An NPU specializes in matrix multiply-accumulate, packs many low-precision units and reuses data on chip, so the energy per operation is orders of magnitude lower. In exchange it is unsuited to non-AI work or unsupported operations.

Which dedicated circuit is built into smartphone SoCs and similar chips to perform AI inference quickly and with low power?

  1. APLL (phase-locked loop)
  2. BADC
  3. CNPU (neural processing unit)
Show answer and explanation

Answer: C. NPU (neural processing unit)

An NPU is a dedicated circuit that executes, in massive parallel, the multiply-accumulate operations dominating neural networks, often at low precision such as 8-bit integers or less. Its performance per watt exceeds a general-purpose CPU's by an order of magnitude or more, letting phones handle photo enhancement, speech recognition and translation on-device. Performance is quoted in TOPS (trillions of operations per second), and NPUs are spreading into PC SoCs.

In AI accelerators and similar, which architecture arranges many multiply-accumulate units in a grid and computes matrix products by passing data from one unit to the next?

  1. ASystolic array
  2. BCrossbar switch
  3. CRing buffer
Show answer and explanation

Answer: A. Systolic array

A systolic array arranges multiply-accumulate units (MACs) in a 2D grid and pumps data rhythmically from neighbor to neighbor—named after the heart's systolic contractions. Each loaded value is reused by many units, cutting memory accesses and making matrix operations efficient. Google's TPU, for example, uses large arrays such as 128×128.

  1. What is the main reason systolic arrays in AI accelerators compute matrix products efficiently?
  2. In AI inference, what is the main benefit of converting (quantizing) weights from FP32 to INT8?
  3. What should be kept in mind about AI accelerator performance figures quoted as "XX TOPS"?

Practice AI Hardware & Software Stack questions

Key terms explained

NPU (neural processing unit)
A dedicated circuit for running neural-network inference quickly and at low power. It is built into the SoCs of smartphones and PCs.
GPU
Short for graphics processing unit: a processor with many simple execution units that processes large amounts of data in parallel. Besides graphics, it is widely used for AI training and inference.
Systolic array
An arrangement of many multiply-accumulate units in a grid that computes matrix products by passing data to neighboring units step by step. It is used in AI accelerators.
INT8
An 8-bit signed integer (−128 to 127). In AI inference, quantizing weights and activations to INT8 with a scale factor greatly cuts computation and memory and allows fast processing on small integer circuits.
TOPS
Trillions of operations per second, a performance figure for AI accelerators, usually a theoretical peak counted at low precision.

Related comparisons