AI Hardware & Software Stack

What is the difference between training and inference for a neural network?

  1. ATraining repeats forward passes, backpropagation and weight updates to determine weights; inference only runs forward passes with fixed weights to produce answers
  2. BBoth training and inference perform only forward passes
  3. CInference updates weights, and training uses no weights
Show answer and explanation

Answer: A. Training repeats forward passes, backpropagation and weight updates to determine weights; inference only runs forward passes with fixed weights to produce answers

Training computes outputs from inputs (the forward pass), propagates the error against the correct answer backward from output to input to obtain each weight's gradient (backpropagation), and adjusts the weights slightly according to the gradients, repeating this over large amounts of data. Inference uses the learned weights and runs only the forward pass. Backpropagation needs about twice the computation of the forward pass and intermediate results must be stored, so training is far heavier than inference.

  • A. Correct. Training needs far more computation and memory.
  • B. Training also performs backpropagation and weight updates.
  • C. It is the other way around.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. Which is NOT a reason training needs much more memory than inference for the same model?
  2. A 7-billion-parameter (7B) model is trained with weights, gradients and Adam's two states all held in FP32 (4 bytes). Excluding activations, roughly how much memory is needed?
  3. If a 7-billion-parameter (7B) model's weights are held in FP16 (2 bytes), roughly how much memory do the weights take?
  4. If the 7B model (about 14 GB in FP16) is quantized to 4-bit integers (INT4), roughly how much memory do the weights take?
  5. How many floating-point operations (counting multiplies and adds separately) does multiplying an M × K matrix by a K × N matrix take, roughly?