AI Hardware & Software Stack

Why can inference runtimes such as TensorRT or ONNX Runtime run faster than using the training framework directly for inference?

  1. AThey optimize the graph for inference only, applying fusion, quantization and hardware-specific kernel selection
  2. BThey skip computation and give approximate answers
  3. CThey increase the number of model weights
Show answer and explanation

Answer: A. They optimize the graph for inference only, applying fusion, quantization and hardware-specific kernel selection

Training frameworks prioritize flexibility and recording for backpropagation, which is wasteful for inference alone. After loading a model, an inference runtime strips parts unneeded for inference, fuses operations, converts weights and activations to lower precision where appropriate and selects the fastest kernels for the target GPU or NPU. This greatly reduces latency and memory for the same model.

  • A. Correct. Backpropagation machinery is no longer needed.
  • B. They do not skip computation.
  • C. The number of weights does not increase.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. Why do element-wise operations such as ReLU or layer normalization tend to take a lot of time relative to their operation count in AI workloads?
  2. Why are convolutions in CNNs sometimes converted into matrix multiplications, for example with im2col?
  3. What is the difference between post-training quantization (PTQ) and quantization-aware training (QAT)?
  4. In post-training quantization of activations to INT8, what is the purpose of calibration?
  5. In quantizing large language models, a few channels with large outliers can hurt accuracy. What is a common countermeasure?