GPU Programming

In GPU mixed-precision computation, why are multiplications often done in a low precision such as FP16 while accumulation is done in FP32?

  1. AMultiplication cannot be done in FP32
  2. BSumming many products in low precision accumulates rounding error and loses small values, so accumulation alone uses higher precision
  3. CAddition cannot be done in FP16
Show answer and explanation

Answer: B. Summing many products in low precision accumulates rounding error and loses small values, so accumulation alone uses higher precision

FP16 has only a 10-bit significand, so adding a small product to a large running sum can round the product away. A matrix product sums thousands of products, so such errors accumulate and skew the result. So the multiplications, which dominate memory and arithmetic cost, are done in low precision, and only the accumulation uses FP32. Most tensor cores perform this kind of multiply-accumulate in a single instruction.

  • A. FP32 multiplication works but is slower and uses more power.
  • B. Correct. It gets both speed and accuracy.
  • C. FP16 can add, but with larger error.
Practice this topic Back to topic

More GPU Programming questions

  1. Summing the same floating-point data on a GPU gives results whose last digits differ slightly from run to run. What is the most likely cause?
  2. What is a common guideline for choosing the number of threads per block in CUDA?
  3. What is the advantage of writing a kernel with a loop like the one above (a grid-stride loop)?
  4. For what kind of access does CUDA constant memory perform especially well?
  5. In general, how do GPU caches compare with CPU caches?