Learning Rate
Definition
The learning rate determines the size of optimization steps when adjusting model parameters.
Too high values can make training unstable; too low values slow it down or lead to suboptimal optimization. Effective step size also depends on the optimizer.
Simply explained
How to set the size of each learning step.
Why it matters in practice
One of the most important hyperparameters for neural training and fine-tuning.
Example from business
A fine-tuning compares multiple learning rates on a fixed validation set.
Limits and pitfalls
- Use test set for tuning; consider learning rate without batch size/optimizer; infer from one run.
Origin & history
Gradient-based optimization has long used a step size; in deep learning, the learning rate is a central hyperparameter.
Use in mid-sized companies
- Possible application in SMEs: test a reasonable range before own fine-tuning and observe training curves.
Frequently asked questions
Is a smaller learning rate always safer?
No, it can be inefficient or suboptimal. Is it learned? In standard training, it is usually predefined or scheduled.
Sources
- Deep Learning, Optimization · Goodfellow et al., Deep Learning, Optimization
Sources checked on 10.10.2026.
Editorially reviewed on: