AI Hardware & Software Stack

When computing softmax exp(xi)/Σexp(xj), why is the maximum often subtracted from each xi first?

  1. ATo approximate the result for speed
  2. BTo force the largest element's output to 0
  3. CThe result is unchanged, and it prevents exp from overflowing to infinity
Show answer and explanation

Answer: C. The result is unchanged, and it prevents exp from overflowing to infinity

exp grows explosively; in FP16, for example, exp(x) exceeds the largest representable value once x passes about 11. Subtracting the same constant m from every x multiplies numerator and denominator by exp(-m), leaving the result unchanged. Subtracting the maximum makes every exponent 0 or less, so exp stays at most 1 and cannot overflow. It is especially important in low-precision AI computation and is used in fast attention implementations.

  • A. The result is exactly the same.
  • B. The largest element gets the biggest output, not 0.
  • C. Correct. Numerator and denominator are scaled by the same factor.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. Which describes the idea of speculative decoding for speeding up LLM inference?
  2. Why are lookups in the large embedding tables used by recommendation models and similar hard for AI hardware?
  3. Which orders the AI software stack from the layer closest to users down to the layer closest to hardware?
  4. What is the difference between training and inference for a neural network?
  5. Which is NOT a reason training needs much more memory than inference for the same model?