AI Hardware & Software Stack

Which is NOT a reason training needs much more memory than inference for the same model?

  1. ATraining multiplies the number of weights several times over
  2. BIntermediate results of the forward pass (activations) must be stored for backpropagation
  3. CGradients and optimizer state, as many as the weights, must be held
Show answer and explanation

Answer: A. Training multiplies the number of weights several times over

The number of weights is the same for training and inference. Training, however, must store each layer's intermediate forward results to compute gradients during backpropagation, and also needs gradients equal in number to the weights plus per-weight values held by optimizers such as Adam (two moving averages, for instance). So training memory is many times that of inference, which needs only the weights, and training large models requires many GPUs.

  • A. Correct. The number of weights is the same for training and inference.
  • B. Stored activations are a large part of training memory.
  • C. Gradients and optimizer state need several times the weight memory.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. A 7-billion-parameter (7B) model is trained with weights, gradients and Adam's two states all held in FP32 (4 bytes). Excluding activations, roughly how much memory is needed?
  2. If a 7-billion-parameter (7B) model's weights are held in FP16 (2 bytes), roughly how much memory do the weights take?
  3. If the 7B model (about 14 GB in FP16) is quantized to 4-bit integers (INT4), roughly how much memory do the weights take?
  4. How many floating-point operations (counting multiplies and adds separately) does multiplying an M × K matrix by a K × N matrix take, roughly?
  5. For the product of two N × N FP16 (2-byte) matrices, assuming each input is read once and the result written once, roughly what is the arithmetic intensity (operations per byte)?