AI Hardware & Software Stack
In quantizing large language models, a few channels with large outliers can hurt accuracy. What is a common countermeasure?
- AAlways zero out channels with outliers
- BReduce the quantization bit width
- CUse scales per channel or per small group instead of one scale for the whole tensor
Show answer and explanation
Answer: C. Use scales per channel or per small group instead of one scale for the whole tensor
Quantizing a whole tensor with a single scale stretches the range to fit a few large outliers, leaving the vast majority of small values with only a few levels. With scales per channel or per group of 64-128 elements, an outlier affects only its own group. Other techniques, such as shifting outliers into the weights before quantizing, are also used.
- A. Essential information is lost and accuracy drops sharply.
- B. Fewer bits would worsen accuracy further.
- C. Correct. It confines the outliers' effect.