Gradient Checkpointing
The short answer · 10 words
In short
Gradient Checkpointing saves training memory by recalculating selected activations later.
Flow-alerta.cab (2026): Gradient Checkpointing. https://flow-alerta.cab/en/glossar/gradient-checkpointing. Editorially reviewed on: 10 October 2026.
How much detail would you like?
Practical
It allows for larger models or batches on the same hardware, but not for free.
Hypothetical example: A Transformer activates checkpointing and compares VRAM, throughput and training time.
Use checkpointing when activation memory is the bottleneck, and compare memory needs and training time.
At a glance
- Core
- Instead of storing all intermediate values for backpropagation, the training only retains checkpoints and recalculates missing activations. Memory decreases, computing time increases.
- Use
- It allows for larger models or batches on the same hardware, but not for free.
- Example
- Hypothetical example: A Transformer activates checkpointing and compares VRAM, throughput and training time.
- Limits
- Confuse with model checkpoints; ignore extra computational costs; overlook incompatible random or stateful operations.
In context
Backpropagation
Efficient method for calculating gradients in multilayer neural networks.
Gradient Accumulation
Gradient accumulation simulates larger effective batches over several smaller forward and reverse runs.
Fully Sharded Data Parallel (FSDP)
FSDP distributes model states across multiple accelerators to train large models in memory-efficient fashion.
Questions about this term
Does it store model stands?
No.
Does training get faster?
Usually slower, but more memory-efficient.
All details and background
Instead of storing all intermediate values for backpropagation, the training only retains checkpoints and recalculates missing activations. Memory decreases, computing time increases.
ELI5 – Explained simply
The model only remembers intermediate stations and calculates paths in between later again.
Why it matters in practice
It allows for larger models or batches on the same hardware, but not for free.
Example
Hypothetical example: A Transformer activates checkpointing and compares VRAM, throughput and training time.
Pitfalls and limits
- Confuse with model checkpoints; ignore extra computational costs; overlook incompatible random or stateful operations.
Origin & history
Chen et al. published “Training Deep Nets with Sublinear Memory Cost” in 2016 to exchange additional computation for lower training memory.
Use in mid-sized companies
- Use checkpointing when activation memory is the bottleneck, and compare memory needs and training time.
Related terms
- Backpropagation
- Activation
- VRAM
- Gradient Accumulation
- FSDP
Matching Flow-alerta service
AI opportunity analysis & roadmapAI consulting: Examine tasks, data and prerequisites of a possible AI use case.Matching tool / use case
View use casesPyTorch torch.utils.checkpoint; documented RNG and condition treatment.Sources
Quellenprüfung: 10.10.2026. Definition und dokumentierte Einordnung abgeglichen; Beispiele beschreiben mögliche Anwendungen.
Editorially reviewed on: