AI Hardware & Software Stack

If a 7-billion-parameter (7B) model's weights are held in FP16 (2 bytes), roughly how much memory do the weights take?

  1. AAbout 14 GB
  2. BAbout 7 GB
  3. CAbout 28 GB
Show answer and explanation

Answer: A. About 14 GB

Weight memory is the number of parameters times the bytes per parameter: 7 billion × 2 bytes = 14 billion bytes, about 14 GB. Inference also needs memory for intermediate results and, for large language models, the KV cache. This calculation is the first estimate of which devices or GPUs can run a model.

  • A. Correct. 7 billion × 2 bytes.
  • B. That is for 1 byte per parameter (INT8).
  • C. That is for FP32 (4 bytes).
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. If the 7B model (about 14 GB in FP16) is quantized to 4-bit integers (INT4), roughly how much memory do the weights take?
  2. How many floating-point operations (counting multiplies and adds separately) does multiplying an M × K matrix by a K × N matrix take, roughly?
  3. For the product of two N × N FP16 (2-byte) matrices, assuming each input is read once and the result written once, roughly what is the arithmetic intensity (operations per byte)?
  4. On an accelerator with 100 TFLOPS peak and 2 TB/s memory bandwidth, a workload with arithmetic intensity 10 FLOP/byte runs. What performance does the Roofline model predict?
  5. Why does a fully connected layer (a weight-matrix-by-vector product) tend to be memory-bound with a single input (batch size 1)?