Computer Arithmetic

When training neural networks in FP16 (half precision: 1 sign, 5 exponent, 10 significand bits), which problem tends to arise and is countered by scaling up the loss (loss scaling)?

  1. AThe significand is too long, increasing memory usage
  2. BThe range is narrow, so small gradients become zero
  3. CValues are treated as integers, so fractions cannot be represented
Show answer and explanation

Answer: B. The range is narrow, so small gradients become zero

With only 5 exponent bits, FP16 represents positive values from about 6 × 10⁻⁸ (including subnormals) to 65504, a narrow range. Many gradients computed during training are very small and underflow to zero in FP16, stalling learning. Loss scaling multiplies the loss by, say, 1024 before backpropagation, lifting gradients into the representable range, and undoes the scaling before updating weights. The significand is 10 bits, shorter than FP32's, and fractions are representable.

  • A. The significand is shorter than FP32's, halving memory use.
  • B. Correct. Its narrow range makes small gradients vanish.
  • C. FP16 is a floating-point format and represents fractions.
Practice this topic Back to topic

More Computer Arithmetic questions

  1. What is the largest finite value representable in FP16 (1 sign, 5 exponent bits with bias 15, 10 significand bits)?
  2. How are the two FP8 formats, E4M3 (4 exponent, 3 significand bits) and E5M2 (5 exponent, 2 significand bits), usually used?
  3. In AI inference, what is the main benefit of converting (quantizing) weights from FP32 to INT8?
  4. Data ranging from −6.35 to 6.35 is quantized to symmetric INT8 (−127 to 127) centered at zero, with scale = maximum absolute value ÷ 127. Which integer represents 1.0?
  5. Why is fixed-point rather than floating-point arithmetic often used in small microcontrollers and DSPs?