Momentum
Definition
Momentum stabilizes the direction of steps during model training.
An optimization method that uses a smoothed trajectory of past gradients to stabilize direction and traverse valleys faster.
Simply explained
Like momentum when rolling, the previous movement influences the next learning step.
Why it matters in practice
Helps realistically assess model quality, limits, and technical decisions.
Example from business
Strongly fluctuating mini-batch gradients are smoothed via an impulse term.
Limits and pitfalls
- Ignoring the interaction of momentum and learning rate or failing to monitor oscillation and overshooting.
Origin & history
Momentum was adopted from numerical optimization into neural network training.
Use in mid-sized companies
- Possible application: Strongly fluctuating mini-batch gradients are smoothed via an impulse term.
Frequently asked questions
What does momentum mean?
An optimization method that uses a smoothed trajectory of past gradients to stabilize direction and traverse valleys faster.
Which mistakes should be avoided?
Ignoring the interaction of momentum and learning rate or failing to monitor oscillation and overshooting.
Sources
Source checked on 10.10.2026. Editorial application examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: