What is AI inference?
Running only the forward pass with weights fixed by training to produce answers from inputs. It needs less computation and memory than training.
Field: AI Hardware & Software Stack
AI inference in the explanations
Game graphics and AI inference are fine with single or half precision, so consumer GPUs include few FP64 units and spend the area on lower precisions.
Why is double-precision (FP64) performance on many consumer…
Questions that mention AI inference (5)
- Which dedicated circuit is built into smartphone SoCs and similar chips to perform AI inference quickly and with low power?
- In AI inference, what is the main benefit of converting (quantizing) weights from FP32 to INT8?
- Which is NOT an advantage of running AI inference on the device itself (at the edge), such as a smartphone, rather than on cloud servers?
- Why are the NPUs in smartphone SoCs more power-efficient than CPUs for AI inference?
- Why is double-precision (FP64) performance on many consumer gaming GPUs much lower than single-precision (FP32), for example one thirty-second?
Related terms
NPU (neural processing unit) TOPS INT8 Branch prediction AI compiler All-reduce Arithmetic intensity Attention mechanism Backpropagation Batch size