What is Backpropagation?

Backpropagation in the explanations

• Loss scaling multiplies the loss by, say, 1024 before backpropagation, lifting gradients into the representable range, and undoes the scaling before updating weights.

  When training neural networks in FP16 (half precision: 1 si…

• Training computes outputs from inputs (the forward pass), propagates the error against the correct answer backward from output to input to obtain each weight's gradient (backpropagation), and adjusts the weights slightly according to the gradients, repeating this over large amounts of data.

  What is the difference between training and inference for a…

• Training, however, must store each layer's intermediate forward results to compute gradients during backpropagation, and also needs gradients equal in number to the weights plus per-weight values held by optimizers such as Adam (two moving averages, for instance).

  Which is NOT a reason training needs much more memory than …

Questions that mention Backpropagation (7)

1. When training neural networks in FP16 (half precision: 1 sign, 5 exponent, 10 significand bits), which problem tends to arise and is countered by scaling up the loss (loss scaling)?
2. What is the difference between training and inference for a neural network?
3. Which is NOT a reason training needs much more memory than inference for the same model?
4. Which describes the computational graph used by AI frameworks and compilers?
5. Why can inference runtimes such as TensorRT or ONNX Runtime run faster than using the training framework directly for inference?
6. Which describes gradient checkpointing (activation recomputation) used in training?
7. For a Transformer with N parameters trained on D tokens, which approximation is commonly used for the number of training operations?

Related terms

Significand Computational graph ONNX Inference runtime GPU NPU (neural processing unit) Gradient checkpointing Batch size AI compiler AI inference