AI Hardware & Software Stack

If the 7B model (about 14 GB in FP16) is quantized to 4-bit integers (INT4), roughly how much memory do the weights take?

  1. AAbout 7 GB
  2. BAbout 1.75 GB
  3. CAbout 3.5 GB
Show answer and explanation

Answer: C. About 3.5 GB

Each parameter goes from 16 bits to 4, so memory falls to a quarter, about 3.5 GB (plus a little for quantization scales and similar metadata). Less memory lets large models run on PCs and smartphones, and reading less data from memory also speeds up inference. Techniques that minimize the loss of accuracy become important.

  • A. That is for INT8 (8 bits).
  • B. That is for 2 bits.
  • C. Correct. 16 bits to 4 bits is one quarter.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. How many floating-point operations (counting multiplies and adds separately) does multiplying an M × K matrix by a K × N matrix take, roughly?
  2. For the product of two N × N FP16 (2-byte) matrices, assuming each input is read once and the result written once, roughly what is the arithmetic intensity (operations per byte)?
  3. On an accelerator with 100 TFLOPS peak and 2 TB/s memory bandwidth, a workload with arithmetic intensity 10 FLOP/byte runs. What performance does the Roofline model predict?
  4. Why does a fully connected layer (a weight-matrix-by-vector product) tend to be memory-bound with a single input (batch size 1)?
  5. Why does increasing the batch size (the number of inputs processed together) tend to improve inference efficiency?