L2 Regularization
Definition
L2 regularization penalizes large parameters via the sum of their squares.
It typically pulls weights continuously toward zero without usually setting them exactly to zero. With optimizers, "weight decay" is not always mathematically identical to L2 regularization.
Simply explained
How to penalize larger control parameters more strongly so that no single weight becomes unnecessarily extreme.
Why it matters in practice
Relevant for controlling overfitting and stabilizing many models.
Example from business
A ridge regression reduces extreme coefficients in correlated features.
Limits and pitfalls
- Equating L2 and weight decay outright; choosing regularization based on test set; ignoring feature scale.
Origin & history
Hoerl and Kennard introduced ridge regression in 1970 as stabilization for multicollinearity.
Use in mid-sized companies
- Possible application in SMEs: compare baselines with and without L2 on validation data.
Frequently asked questions
Does L2 set weights exactly to zero?
Typically not. Is weight decay always L2? No, the decoupling was recently discussed with AdamW.
Sources
- Ridge Regression · Hoerl & Kennard (1970), Ridge Regression
Sources checked on 10.10.2026.
Editorially reviewed on: