What is Gradient checkpointing?
Storing only some intermediate forward results and recomputing the rest during backpropagation, trading extra computation for less memory.
Field: AI Hardware & Software Stack
Gradient checkpointing in the explanations
Gradient checkpointing stores results only every few layers, and when backpropagation reaches a segment it reruns the forward pass from the saved point to regenerate the needed values.
Which describes gradient checkpointing (activation recomput…
Questions that mention Gradient checkpointing (1)
Related terms
Backpropagation Batch size AI compiler AI inference All-reduce Arithmetic intensity Attention mechanism Computational graph Data parallelism FlashAttention