AI Hardware & Software Stack
Which describes gradient checkpointing (activation recomputation) used in training?
- AStore only some intermediate forward results and recompute the rest when needed during backpropagation, trading extra computation for less memory
- BSaving model weights to disk partway through training
- CTraining without computing gradients
Show answer and explanation
Answer: A. Store only some intermediate forward results and recompute the rest when needed during backpropagation, trading extra computation for less memory
Backpropagation needs each layer's intermediate forward results, but storing them all makes memory grow with layer count and batch size. Gradient checkpointing stores results only every few layers, and when backpropagation reaches a segment it reruns the forward pass from the saved point to regenerate the needed values. At the cost of roughly one extra forward pass, activation memory is greatly reduced.
- A. Correct. It is commonly used when memory runs short.
- B. It is different from saving training checkpoints.
- C. Gradients are still computed.