Zero Redundancy Optimizer (ZeRO)
Definition
ZeRO distributes training states across multiple processes, thereby reducing duplicated memory usage.
A family of techniques that partitions optimizer states, gradients, and parameters across data-parallel processes to reduce memory redundancy.
Simply explained
Multiple computing devices share the required learning information instead of each device holding all information completely.
Why it matters in practice
The term helps to select, develop, evaluate AI systems in a well-founded manner and realistically assess their technical risks.
Example from business
A team uses ZeRO Stage 2 and measures memory, communication, and restart on multiple GPUs.
Limits and pitfalls
- ZeRO is not zero-overhead; stages differ, communication and checkpointing become more complex.
Origin & history
Rajbhandari and co-authors published ZeRO in 2019 as a foundation for very large training runs.
Use in mid-sized companies
- Possible example: A team uses ZeRO Stage 2 and measures memory, communication, and restart on multiple GPUs.
Frequently asked questions
What does Zero Redundancy Optimizer (ZeRO) mean?
A family of techniques that partitions optimizer states, gradients, and parameters across data-parallel processes to reduce memory redundancy. What should be considered? ZeRO is not zero-overhead; stages differ, communication and checkpointing become more complex.
Sources
- ZeRO · Rajbhandari et al., ZeRO
Source checked on 10.10.2026. Editorial application examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: