AI Hardware & Software Stack

For the product of two N × N FP16 (2-byte) matrices, assuming each input is read once and the result written once, roughly what is the arithmetic intensity (operations per byte)?

  1. A1/3
  2. BN/3
  3. C3N
Show answer and explanation

Answer: B. N/3

The operation count is 2N³, and memory traffic is 3N² elements for the three matrices, or 6N² bytes in FP16. Arithmetic intensity is 2N³ ÷ 6N² = N/3, about 340 for N = 1024. The larger a matrix product, the more computation per byte, so it easily exploits arithmetic performance. By contrast, element-wise addition has intensity below 1 and is limited by memory bandwidth.

  • A. It grows in proportion to N.
  • B. Correct. 2N³ ÷ 6N² = N/3.
  • C. That overestimates the operation count.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. On an accelerator with 100 TFLOPS peak and 2 TB/s memory bandwidth, a workload with arithmetic intensity 10 FLOP/byte runs. What performance does the Roofline model predict?
  2. Why does a fully connected layer (a weight-matrix-by-vector product) tend to be memory-bound with a single input (batch size 1)?
  3. Why does increasing the batch size (the number of inputs processed together) tend to improve inference efficiency?
  4. What is a drawback of making the inference batch size too large?
  5. LLM inference splits into a stage that processes the input prompt together (prefill) and a stage that generates output one token at a time (decode). With small batches, what mainly limits each stage?