Weight Decay
Definition
Weight decay reduces model weights during optimization and serves as regularization.
A regularization that shrinks parameters toward zero during optimization steps; with SGD it often corresponds to L2 regularization.
Simply explained
During learning, the model’s weight values are slightly pulled toward zero. This counteracts very large values.
Why it matters in practice
The term helps to select, develop, evaluate AI systems in a well-founded manner and realistically assess their technical risks.
Example from business
A team tunes weight decay on validation data and deliberately excludes bias and norm parameters.
Limits and pitfalls
- With adaptive optimizers, coupled L2 penalty is not identical; too strong decay leads to underfitting.
Origin & history
Weight decay has been common since early neural nets; AdamW decoupled it from adaptive gradients in 2017.
Use in mid-sized companies
- Possible example: A team tunes weight decay on validation data and deliberately excludes bias and norm parameters.
Frequently asked questions
What does weight decay mean?
A regularization that shrinks parameters toward zero during optimization steps; with SGD it often corresponds to L2 regularization. What should you watch out for? With adaptive optimizers, coupled L2 penalty is not identical; too strong decay leads to underfitting.
Sources
- Decoupled Weight Decay Regularization · Loshchilov and Hutter, Decoupled Weight Decay Regularization
Source checked on 10.10.2026. Editorial usage examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: