AI Hardware & Software Stack

Why are convolutions in CNNs sometimes converted into matrix multiplications, for example with im2col?

  1. AConversion reduces the operation count tenfold
  2. BSo that highly optimized matrix-multiplication libraries and matrix units such as tensor cores can be used
  3. CConversion always reduces memory usage
Show answer and explanation

Answer: B. So that highly optimized matrix-multiplication libraries and matrix units such as tensor cores can be used

im2col builds a large matrix in which the input values of each convolution window are laid out as a column, then computes the convolution as a product with a matrix of filters. Overlapping windows duplicate input values and increase memory, but decades-optimized matrix-multiplication libraries and matrix units such as tensor cores can be used directly, so it is often faster overall. Some implementations achieve the same effect without materializing the duplicates.

  • A. The operation count is unchanged.
  • B. Correct. Rearranging the input increases memory, though.
  • C. im2col duplicates input data, so memory actually increases.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. What is the difference between post-training quantization (PTQ) and quantization-aware training (QAT)?
  2. In post-training quantization of activations to INT8, what is the purpose of calibration?
  3. In quantizing large language models, a few channels with large outliers can hurt accuracy. What is a common countermeasure?
  4. In LLM inference, why does weight-only 4-bit quantization (W4A16, activations kept in FP16) especially speed up decode?
  5. Which statement about pruning neural networks is correct?