AI Hardware & Software Stack

A matrix product followed by a bias add and a ReLU is fused from three separate kernels into one. What mainly decreases?

  1. AThe number of multiplications
  2. BThe amount of memory (HBM) traffic and the number of kernel launches
  3. CThe number of model weights
Show answer and explanation

Answer: B. The amount of memory (HBM) traffic and the number of kernel launches

With separate kernels, the matrix product writes its result to memory, the bias kernel reads and writes it, and the ReLU kernel reads and writes it again. Fused, the bias and ReLU are applied right where the product is computed and only the final result is written, eliminating two round trips of a large intermediate tensor and reducing launch overhead. The amount of computation itself is unchanged.

  • A. The matrix product's multiplications are unchanged.
  • B. Correct. The amount of computation is unchanged.
  • C. The number of weights is unchanged.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. What do AI compilers such as XLA, TVM or PyTorch's compile feature mainly do?
  2. Which describes the computational graph used by AI frameworks and compilers?
  3. Which describes ONNX?
  4. Why can inference runtimes such as TensorRT or ONNX Runtime run faster than using the training framework directly for inference?
  5. Why do element-wise operations such as ReLU or layer normalization tend to take a lot of time relative to their operation count in AI workloads?