AI Hardware & Software Stack
In LLM inference, why does weight-only 4-bit quantization (W4A16, activations kept in FP16) especially speed up decode?
- A4-bit multiplication is more accurate than 16-bit
- BDecode is memory-bound, so cutting weight reads to a quarter translates directly into speed
- CKeeping activations in FP16 reduces the operation count
Show answer and explanation
Answer: B. Decode is memory-bound, so cutting weight reads to a quarter translates directly into speed
Decode reads all the weights for every token, so its speed is set by memory bandwidth. With 4-bit weights the data read falls to a quarter, and even if they are expanded to FP16 on chip before computing, total time drops sharply. For compute-bound prefill the gain is small, and quantization that performs the computation itself in low precision helps more. Which quantization pays off depends on where the bottleneck is.
- A. Precision is lower.
- B. Correct. Even if computation is done in FP16 after expansion, reads shrink.
- C. The operation count is unchanged.