AI Hardware & Software Stack

What should be kept in mind about AI accelerator performance figures quoted as "XX TOPS"?

  1. AIt is usually a theoretical peak counted at a low precision such as INT8, and real model speed varies greatly with memory bandwidth and utilization
  2. BTwice the TOPS always means twice the speed for any model
  3. CTOPS represents memory capacity
Show answer and explanation

Answer: A. It is usually a theoretical peak counted at a low precision such as INT8, and real model speed varies greatly with memory bandwidth and utilization

TOPS (trillions of operations per second) is the theoretical figure with every unit busy all the time, usually counted at INT8 or lower and sometimes assuming sparsity. Real models may be limited by memory bandwidth, leave units idle or fall back to other circuits for unsupported operations, so effective performance can be much lower. Comparing products requires measured results under the same conditions.

  • A. Correct. Check the precision assumed and measured results.
  • B. For memory-bound models, more TOPS does not help.
  • C. It stands for trillions of operations per second.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. Which describes MFU (Model FLOPs Utilization), a metric of large-scale training efficiency?
  2. For a Transformer with N parameters trained on D tokens, which approximation is commonly used for the number of training operations?
  3. For an LLM with N parameters, roughly how many operations does generating one token take in inference (excluding attention)?
  4. When computing softmax exp(xi)/Σexp(xj), why is the maximum often subtracted from each xi first?
  5. Which describes the idea of speculative decoding for speeding up LLM inference?