AI Hardware & Software Stack

Why are the NPUs in smartphone SoCs more power-efficient than CPUs for AI inference?

  1. ANPUs compute with light instead of electricity
  2. BThey pack many low-precision (such as INT8) multiply-accumulate units and omit general-purpose circuitry unneeded for AI
  3. CNPUs always run at higher clock speeds than CPUs
Show answer and explanation

Answer: B. They pack many low-precision (such as INT8) multiply-accumulate units and omit general-purpose circuitry unneeded for AI

To run anything, a CPU exercises much circuitry per instruction, such as branch prediction and instruction reordering. An NPU specializes in matrix multiply-accumulate, packs many low-precision units and reuses data on chip, so the energy per operation is orders of magnitude lower. In exchange it is unsuited to non-AI work or unsupported operations.

  • A. Typical NPUs are electronic circuits.
  • B. Correct. Specialization raises operations per watt.
  • C. Clocks are often lower, if anything.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. What should be kept in mind about AI accelerator performance figures quoted as "XX TOPS"?
  2. Which describes MFU (Model FLOPs Utilization), a metric of large-scale training efficiency?
  3. For a Transformer with N parameters trained on D tokens, which approximation is commonly used for the number of training operations?
  4. For an LLM with N parameters, roughly how many operations does generating one token take in inference (excluding attention)?
  5. When computing softmax exp(xi)/Σexp(xj), why is the maximum often subtracted from each xi first?