What Is an NPU? The AI Engine in Phones and PCs Explained

In everyday life

Terms like "AI PC" and "with NPU" now show up in PC stores and on spec sheets. In May 2024, Microsoft announced a new class of PCs called "Copilot+ PCs." One requirement is an NPU that can do 40 trillion or more operations per second. iPhone spec sheets also list items such as a "16-core Neural Engine."

Think of removing a stranger from a photo, turning a recording into text, or blurring the background in a video call. Behind features like these, the NPU is often the part doing the work. This article looks at what an NPU does, step by step. It also asks why companies go to the trouble of building a dedicated circuit for it.

An NPU's job: lots of "multiply, then add"

Most AI today is built on a set of calculation steps called a neural network. Inside, it does one thing a vast number of times. It multiplies input numbers by fixed numbers (weights) and adds up the results.

One multiply, plus one add that sums in its result, is called a multiply-accumulate operation. It is often shortened to MAC. "Matrix Multiplication FLOPs and Arithmetic Intensity" shows how to count the MACs an AI needs.

An NPU is a part made just for AI math, with many multiply-accumulate circuits side by side. Its name means "a unit that processes neural networks." It is usually not a separate chip. Instead, it is often built into the one chip that holds a device's main circuits, along with the CPU and GPU. Such a chip is called a system on chip (SoC) ("What Chips Are Inside a Smartphone?").

Companies use different names for it. Apple calls it the Neural Engine, and Qualcomm calls it the Hexagon NPU. In this article, "NPU" means any circuit made just for AI inside a phone or PC.

Why a dedicated circuit, not the CPU or GPU?

A CPU or a GPU can do AI math too. The problem is power. Phones and laptops run on batteries, and a phone with no fan has limited ways to shed heat. So what matters is how much math you get for the same power. This is called performance per watt.

A CPU can handle anything, but it spends a lot of power on things other than the math itself. Its circuits that fetch each instruction, decode it and decide what to do next are always running ("What Is a CPU?"). An NPU drops much of this machinery and fixes the flow of calculation in advance. An AI model (the weights and the calculation steps, bundled together) is first converted into steps for that NPU. Only then is it run.

Trick 1: Use shorter numbers

In AI math, making the numbers a little less precise often barely changes the answer. So instead of 32-bit numbers with a decimal point, NPUs use 8-bit integers. (A 32-bit number is a string of thirty-two 0s and 1s.) This 8-bit format is called INT8. It can represent 256 whole numbers, from -128 to 127.

Each original number is replaced with the nearest of these 256 levels. In AI, this replacement is called "quantization." A multiplying circuit gets smaller and uses less power when it handles fewer digits. In an estimate published in 2014, an 8-bit integer multiply took about one-eighteenth of the energy of a 32-bit decimal multiply.

Some products also use 16-bit numbers with a decimal point (FP16), or even shorter 4-bit integers. "FP32 vs. FP16 vs. BF16 vs. INT8" compares these number formats in a table.

Trick 2: Move less data

In fact, moving numbers takes more power than the math itself. By estimates from the same time, reading one number from memory outside the chip (DRAM) takes over 100 times the energy of one multiply. So an NPU keeps a small memory right next to its calculation circuits. It reuses each number it brings in many times.

In AI math, the same weights can be used at many places in an image, or for many people's inputs. The NPU computes everything it can with the numbers in its nearby memory before moving on. That cuts round trips to the faraway memory. "What Is HBM?" also explains how much the speed of moving data from memory matters in AI.

How it differs from a GPU

A GPU is also a part with many calculation circuits side by side. But a GPU takes on a wide range of jobs, depending on its program: drawing graphics, training AI, scientific computing and more. That flexibility costs power. Some data-center GPUs use several hundred watts each.

NPUs in phones and PCs focus mainly on one job. They use weights already set by training to produce answers for new inputs. This is called inference. "Training," which sets the weights, is usually done on GPUs and similar parts in data centers. "CPU vs. GPU vs. NPU" and "AI training vs. inference" compare these in tables.

Why run AI on the device?

AI math can also be done in a data center across the internet ("What Is Cloud Computing?"). Still, there are reasons to run it on the device. Google's developer documentation lists these as the main benefits:

• Data such as photos and voice does not have to leave the device
• It works without an internet connection
• There is no waiting for data to travel back and forth over the network
• It lowers computing costs for the company that provides the service

Running AI on the device in your hand, not in a data center, is called "edge AI" or "on-device AI." The "edge" means devices at the edge of the network. To run a long time on a battery, an NPU with high performance per watt is a good fit.

Where are NPUs found?

Phones

In 2017, several phone SoCs with circuits made just for AI were announced. In September, Huawei announced the Kirin 970 and said it had a dedicated NPU. The A11 Bionic, which Apple announced the same month, had the company's first Neural Engine. Today, many phone SoCs have AI circuits, including Apple's A series and Qualcomm's Snapdragon. Google has also used its own SoC, called "Tensor," since the Pixel 6 in 2021.

PCs

Apple has put a Neural Engine in its Mac SoCs since the M1 (2020). For Windows PCs, the Copilot+ PCs mentioned above helped spread NPUs. The first Copilot+ PCs, released in 2024, used Qualcomm's Snapdragon X series. Qualcomm's specs rate their NPU at 45 TOPS. Products with AMD and Intel chips were added later.

Data centers

In May 2016, Google revealed the "TPU," a chip it had built in-house for AI. At the time, Google said it had already used TPUs in its data centers for over a year. The TPU is also a chip specialized in matrix multiply-accumulate. But it is a large part for data centers, and from the second generation (2017) it has also been used for training. Parts like this are often called "AI accelerators" (dedicated parts that speed up AI math). The name NPU is used mainly for circuits inside phones and PCs. "Why Are GPUs Used for AI?" introduces how they work.

What is TOPS? How to read the number

An NPU's performance is often given as how many trillion operations it can do in one second. This unit is called TOPS. The T stands for tera (trillion), and OPS stands for operations per second. So 40 TOPS means 40 trillion operations per second.

This number is usually a theoretical maximum. Take the number of multiply-accumulate circuits and multiply it by 2, since the multiply and the add count as two operations. Then multiply by how many times per second the circuits run. It assumes every circuit works nonstop. Real AI almost never gets there.

On top of that, the same chip can show different numbers, depending on how they are counted. The main conditions are:

• Number format: Shorter numbers allow more math at once. So counting with 4-bit numbers can give a bigger figure than counting with 8-bit ones. Counting with 16-bit decimal numbers gives a smaller one.
• Skipping zeros: Some figures skip the math for weights that are zero, yet count the skipped part as "done." Having many zeros is called "sparse," so these are also called sparse figures.
• Moving speed: TOPS does not include how fast numbers can be moved from memory. If moving is slow, the circuits just wait for numbers.
• Software support: Math the NPU does not support is passed to the CPU or other parts.

So you cannot compare NPUs from different makers or generations by TOPS alone. To compare them, the reliable way is to run the apps or models you want to use and look at the measured results (benchmarks).

Common jobs for NPUs

On phones, NPUs are often used for photo processing. Examples are recognizing objects and people in a shot, reducing graininess in low-light photos, and blurring the background. They also handle voice and language, such as turning speech into text and translating between languages. On PCs, one example is blurring the camera background in video calls. Windows Studio Effects is a Windows feature made for PCs with an NPU.

Recently, small versions of text-writing AI have started to run on devices ("How Do Large Language Models Work?"). In 2024, Apple said it runs a model with about 3 billion parameters on the device. Parameters are numbers set by training, such as weights. But which features run on the NPU differs by product and app.

The limits of NPUs

Unused unless the software supports it

NPU designs differ by maker and model. So software is needed to load a trained model, convert it into a form that runs on that hardware, and run it. This software is called an inference runtime. If an app or the OS is not built to use the NPU, the NPU does nothing.

Large models do not fit

The memory next to an NPU is small. So a model's weights are normally kept in the device's main memory. A phone's memory is 8 GB in the iPhone 16 Pro (2024), for example, according to a teardown. That memory is shared with the OS and other apps ("Memory vs Storage: What's the Difference?"). Large models that run in data centers do not fit as they are. For devices, makers choose models with fewer parameters, and shrink them by shortening the numbers.

Common misconceptions

By the numbers

The top four rows are counted under different conditions, so they are not meant to be compared with each other. The two energy rows are estimates Horowitz gave in 2014, for an old manufacturing generation called 45 nm. (They come from tables in papers by Han et al. and Sze et al.) The values themselves change as manufacturing advances. Even so, Sze and colleagues' 2017 tutorial uses one fact as the starting point for AI chip design. Moving numbers costs more energy than computing with them.

Let's roughly estimate the effect of moving speed. Say a text AI with 3 billion weights runs with each weight shortened to 4 bits. With 8 bits to a byte, each weight is 0.5 bytes. The weights total about 1.5 GB. A text AI reads almost all of its weights each time it produces one word (strictly, one token).

Take a chip that can move 135 GB per second from memory (the Snapdragon X Elite's spec). Even so, one full read takes about 0.011 seconds. That caps it at about 90 words per second. The math, meanwhile, is two operations per weight (a multiply and an add), or about 6 billion per word. At 45 TOPS, that takes about 0.00013 seconds, about one-eightieth of the read time. In this case, doubling TOPS would barely change the speed. To keep things simple, we left out reading and writing other data, and differences in number formats.

Learn more

• Compare in a table: CPU vs. GPU vs. NPU, AI training vs. inference, DDR vs. LPDDR vs. GDDR vs. HBM
• Try the numbers: the memory bandwidth calculator (works out how much data memory can move per second)
• Terms for the next step: post-training quantization (shortening a model's numbers after training), structured sparsity (arranging zeros in a fixed pattern to skip math), systolic array (one way to lay out multiply-accumulate circuits)
• Count the math: "Matrix Multiplication FLOPs and Arithmetic Intensity"
• Related articles: "Why Are GPUs Used for AI?", "What Is HBM?"