AI Hardware & Software Stack

What is the difference between post-training quantization (PTQ) and quantization-aware training (QAT)?

  1. APTQ quantizes before training, QAT after
  2. BPTQ quantizes a trained model as is; QAT simulates quantization during training so the weights adapt and errors stay small at low precision
  3. CNeither PTQ nor QAT actually quantizes
Show answer and explanation

Answer: B. PTQ quantizes a trained model as is; QAT simulates quantization during training so the weights adapt and errors stay small at low precision

Post-training quantization converts a trained model's weights (and activations) to lower precision after examining value ranges with a small representative dataset. It is easy but loses accuracy as bit width shrinks. Quantization-aware training simulates quantization rounding in the forward pass during training and adjusts weights with that error included, preserving accuracy even at precisions as low as 4 bits.

  • A. PTQ happens after training.
  • B. Correct. QAT costs more effort but preserves accuracy better.
  • C. Both are quantization methods.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. In post-training quantization of activations to INT8, what is the purpose of calibration?
  2. In quantizing large language models, a few channels with large outliers can hurt accuracy. What is a common countermeasure?
  3. In LLM inference, why does weight-only 4-bit quantization (W4A16, activations kept in FP16) especially speed up decode?
  4. Which statement about pruning neural networks is correct?
  5. Which describes the 2:4 structured sparsity supported by GPU matrix units?