A 16-bit floating-point format with 1 sign, 8 exponent and 7 significand bits. Sharing FP32's exponent width gives it a wide range, so despite low precision it is used for AI training. FP16, with 5 exponent and 10 significand bits, is more precise but has a narrow range.
Field: Computer Arithmetic
bfloat16 is a 16-bit format with 1 sign, 8 exponent and 7 significand bits.
What is the main reason bfloat16 is used for AI training?FP16 and bfloat16 have short significands, so if an update is too small relative to the weight, the rounded sum equals the old weight and the update vanishes (absorption).
In mixed-precision AI training, why are master weights kept…Significand Arithmetic shift Booth's algorithm Carry-lookahead adder Carry-save adder Catastrophic cancellation Exponent field Fixed point Floating-point absorption FP8