AI Hardware & Software Stack

A model with about 14 GB of FP16 weights decodes at batch size 1 on a GPU with 2 TB/s memory bandwidth. What is closest to the lower bound on time per generated token?

  1. AAbout 7 µs
  2. BAbout 7 s
  3. CAbout 7 ms
Show answer and explanation

Answer: C. About 7 ms

Decode must read all weights for every token, so each token takes at least model size ÷ memory bandwidth: 14 GB ÷ 2 TB/s = 0.007 s = 7 ms, a maximum of about 140 tokens per second. Quantizing weights to 4 bits, about 3.5 GB, lowers this bound to about 1.75 ms. This is why weight quantization is especially effective for decode.

  • A. That is off by a factor of 1000.
  • B. That is off by a factor of 1000.
  • C. Correct. 14 GB ÷ 2 TB/s = 7 ms.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. How does the computation of Transformer self-attention grow with input sequence length n?
  2. What is the main idea behind fast attention implementations such as FlashAttention?
  3. What is the main purpose of operator fusion in AI computation?
  4. A matrix product followed by a bias add and a ReLU is fused from three separate kernels into one. What mainly decreases?
  5. What do AI compilers such as XLA, TVM or PyTorch's compile feature mainly do?