What is Inference runtime?

Software that loads a trained model and runs inference quickly through graph optimization, quantization and hardware-specific kernel selection.

Field: AI Hardware & Software Stack

Inference runtime in the explanations

After loading a model, an inference runtime strips parts unneeded for inference, fuses operations, converts weights and activations to lower precision where appropriate and selects the fastest kernels for the target GPU or NPU.
Why can inference runtimes such as TensorRT or ONNX Runtime…

Questions that mention Inference runtime (1)

Why can inference runtimes such as TensorRT or ONNX Runtime run faster than using the training framework directly for inference?

Related terms

ONNX Backpropagation GPU NPU (neural processing unit) AI compiler AI inference All-reduce Arithmetic intensity Attention mechanism Batch size

Back to the glossary