Computer Arithmetic

What is the advantage of computing a × b + c with an FMA (fused multiply-add) instruction?

  1. AOnly one rounding, so it is more accurate and faster
  2. BThe product is always made an integer before adding, so no error occurs at all
  3. CThe addition is skipped and only the multiplication is done, halving the work
Show answer and explanation

Answer: A. Only one rounding, so it is more accurate and faster

Separate multiply and add instructions round a × b and then round again after adding c, two roundings in all. FMA keeps the unrounded product internally, adds c and rounds just once, so the error is smaller. Being one instruction, it is also faster, and it is used heavily in matrix products, polynomial evaluation and AI. It neither makes the product an integer nor skips the addition. Because the rounding count differs, results can differ slightly from separate operations.

  • A. Correct. A single rounding makes it more accurate and faster.
  • B. The product is not converted to an integer.
  • C. The addition is performed within the same instruction.
Practice this topic Back to topic

More Computer Arithmetic questions

  1. What is the main reason bfloat16 is used for AI training?
  2. When training neural networks in FP16 (half precision: 1 sign, 5 exponent, 10 significand bits), which problem tends to arise and is countered by scaling up the loss (loss scaling)?
  3. What is the largest finite value representable in FP16 (1 sign, 5 exponent bits with bias 15, 10 significand bits)?
  4. How are the two FP8 formats, E4M3 (4 exponent, 3 significand bits) and E5M2 (5 exponent, 2 significand bits), usually used?
  5. In AI inference, what is the main benefit of converting (quantizing) weights from FP32 to INT8?