AI Hardware & Software Stack

How many floating-point operations (counting multiplies and adds separately) does multiplying an M × K matrix by a K × N matrix take, roughly?

  1. AMNK
  2. BM + N + K
  3. C2MNK
Show answer and explanation

Answer: C. 2MNK

The result has M × N elements, each an inner product of length K requiring K multiplications and about K additions, about 2MNK operations in all. AI workloads are often estimated with this formula, and performance is expressed in operations per second (FLOPS). Counting a multiply-add as one operation halves it to MNK, so it matters which convention is used.

  • A. That counts each multiply-add as one.
  • B. The operation count is far larger.
  • C. Correct. Each element needs K multiplies and adds.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. For the product of two N × N FP16 (2-byte) matrices, assuming each input is read once and the result written once, roughly what is the arithmetic intensity (operations per byte)?
  2. On an accelerator with 100 TFLOPS peak and 2 TB/s memory bandwidth, a workload with arithmetic intensity 10 FLOP/byte runs. What performance does the Roofline model predict?
  3. Why does a fully connected layer (a weight-matrix-by-vector product) tend to be memory-bound with a single input (batch size 1)?
  4. Why does increasing the batch size (the number of inputs processed together) tend to improve inference efficiency?
  5. What is a drawback of making the inference batch size too large?