AI Hardware & Software Stack

Why do element-wise operations such as ReLU or layer normalization tend to take a lot of time relative to their operation count in AI workloads?

  1. AReLU involves more operations than matrix multiplication
  2. BThey do little computation relative to the data read and written per element, so memory bandwidth sets their speed
  3. CGPUs cannot execute element-wise operations
Show answer and explanation

Answer: B. They do little computation relative to the data read and written per element, so memory bandwidth sets their speed

ReLU just reads each element, compares it with 0 and writes it back, very little computation per element. With such low arithmetic intensity, time is set by memory bandwidth rather than compute. They can be a small share of operations yet a noticeable share of time. Fusing them right after matrix products, or combining normalization steps into one kernel, reduces memory traffic.

  • A. They need far fewer operations than matrix products.
  • B. Correct. They are prime candidates for fusion.
  • C. GPUs execute them fine.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. Why are convolutions in CNNs sometimes converted into matrix multiplications, for example with im2col?
  2. What is the difference between post-training quantization (PTQ) and quantization-aware training (QAT)?
  3. In post-training quantization of activations to INT8, what is the purpose of calibration?
  4. In quantizing large language models, a few channels with large outliers can hurt accuracy. What is a common countermeasure?
  5. In LLM inference, why does weight-only 4-bit quantization (W4A16, activations kept in FP16) especially speed up decode?