Computer Arithmetic

In mixed-precision AI training, why are master weights kept in FP32 while most computation is done in FP16 or bfloat16?

  1. AWhen small updates are added to weights, a low-precision format would lose them to absorption
  2. BFP32 computes faster than FP16, so all weight calculations are left to FP32
  3. CFP16 and bfloat16 values are not allowed to be stored in memory
Show answer and explanation

Answer: A. When small updates are added to weights, a low-precision format would lose them to absorption

Training repeatedly adds very small amounts, learning rate × gradient, to the weights. FP16 and bfloat16 have short significands, so if an update is too small relative to the weight, the rounded sum equals the old weight and the update vanishes (absorption). Weights are therefore kept in FP32 so updates accumulate correctly, while lower-precision copies are used for the forward and backward passes. Lower precision computes faster, and low-precision values can certainly be stored.

  • A. Correct. It prevents small updates from vanishing through absorption.
  • B. Lower precision computes faster.
  • C. Low-precision values can be stored in memory.
Practice this topic Back to topic

More Computer Arithmetic questions

  1. In unsigned fixed point with the low 4 of 8 bits as the fraction, what value does the bit pattern 0011 1000 represent?
  2. Which three parts make up an IEEE 754 floating-point number?
  3. How are the bits of an IEEE 754 single-precision (32-bit) number allocated among sign, exponent and significand?
  4. In IEEE 754 double precision (64-bit), how many bits are the exponent and significand?
  5. In IEEE 754 single precision, if the stored exponent field is 130, what is the actual exponent (power of two)?