AI Hardware & Software Stack

What is the main purpose of operator fusion in AI computation?

  1. ATo combine several operations into one kernel, avoiding writing intermediate results to memory and reading them back
  2. BTo merge two AI models into one for higher accuracy
  3. CTo physically fuse arithmetic units together
Show answer and explanation

Answer: A. To combine several operations into one kernel, avoiding writing intermediate results to memory and reading them back

A matrix product is often followed by element-wise operations such as adding a bias, an activation function and normalization. Doing each in a separate kernel means writing intermediate results to memory and reading them back every time, which takes longer than the computation. Combining them into one kernel lets the intermediate values stay in registers or on-chip memory throughout.

  • A. Correct. It especially helps memory-bound operations.
  • B. It is not about merging models.
  • C. It is not physical processing.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. A matrix product followed by a bias add and a ReLU is fused from three separate kernels into one. What mainly decreases?
  2. What do AI compilers such as XLA, TVM or PyTorch's compile feature mainly do?
  3. Which describes the computational graph used by AI frameworks and compilers?
  4. Which describes ONNX?
  5. Why can inference runtimes such as TensorRT or ONNX Runtime run faster than using the training framework directly for inference?