Learning Rate Schedule
Definition
A learning rate schedule changes the learning rate during training according to a fixed or adaptive rule.
Typical strategies are Warmup, Step Decay, Cosine Decay, or plateau-based adjustment. The schedule influences stability and convergence.
Simply explained
How to deliberately change the learning rate over time.
Why it matters in practice
Relevant for stable and efficient training.
Example from business
A Transformer starts with Warmup and then gradually lowers the learning rate stepwise.
Limits and pitfalls
- Copy schedule independently of total training duration; do not tune Warmup and batch size together.
Origin & history
Learning-rate schedules have long been part of neural network optimization; modern Transformers often use Warmup and Decay variants.
Use in mid-sized companies
- Possible application in SMEs: Validate training schedule together with optimizer, batch size, and runtime.
Frequently asked questions
Is a constant learning rate always bad?
No. What is Warmup? An initially increasing learning rate.
Sources
- How to adjust learning rate · PyTorch 2.9, How to adjust learning rate
Sources checked on 12.10.2026.
Editorially reviewed on: