AI Hardware & Software Stack

A 7-billion-parameter (7B) model is trained with weights, gradients and Adam's two states all held in FP32 (4 bytes). Excluding activations, roughly how much memory is needed?

  1. AAbout 28 GB
  2. BAbout 112 GB
  3. CAbout 14 GB
Show answer and explanation

Answer: B. About 112 GB

Per parameter there are 4 bytes for the weight, 4 for the gradient and 4 each for Adam's first and second moving averages, 16 bytes in total. 7 billion × 16 bytes = 112 billion bytes, about 112 GB, before adding activation memory. This often does not fit on one GPU, so techniques such as sharding optimizer state across GPUs or keeping it in lower precision are used.

  • A. That is the weights alone.
  • B. Correct. 16 bytes per parameter.
  • C. That is FP16 weights alone.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. If a 7-billion-parameter (7B) model's weights are held in FP16 (2 bytes), roughly how much memory do the weights take?
  2. If the 7B model (about 14 GB in FP16) is quantized to 4-bit integers (INT4), roughly how much memory do the weights take?
  3. How many floating-point operations (counting multiplies and adds separately) does multiplying an M × K matrix by a K × N matrix take, roughly?
  4. For the product of two N × N FP16 (2-byte) matrices, assuming each input is read once and the result written once, roughly what is the arithmetic intensity (operations per byte)?
  5. On an accelerator with 100 TFLOPS peak and 2 TB/s memory bandwidth, a workload with arithmetic intensity 10 FLOP/byte runs. What performance does the Roofline model predict?