AdamW
The short answer · 17 words
In short
AdamW is a variant of Adam in which Weight Decay is decoupled from the gradient-based optimization step.
Flow-alerta.cab (2026): AdamW. https://flow-alerta.cab/en/glossar/adamw. Editorially reviewed on: 09 October 2026.
How much detail would you like?
Practical
Relevant for own model training, if Weight Decay is to be controlled separately from adaptive gradient optimization.
Possible example: When fine-tuning a Transformer model, a team uses AdamW and adjusts learning rate and weight decay separately using a validation set.
Possible applications: relevant for own model training and fine tuning, especially for Transformer-based models. Companies can thus make training recipes more reproducible, but must validate the effect on the basis of their own data.
At a glance
- Core
- In adaptive optimizers, classical L2 regularization is not generally equivalent to weight decay. AdamW therefore applies weight decay separately from the Adam update and makes regularization controllable separately.
- Use
- Relevant for own model training, if Weight Decay is to be controlled separately from adaptive gradient optimization.
- Example
- Possible example: When fine-tuning a Transformer model, a team uses AdamW and adjusts learning rate and weight decay separately using a validation set.
- Limits
- Treat AdamW merely as a new name for Adam; equate weight decay and L2 regularization in adaptive optimizers; apply weight decay unchecked to all parameters; adopt hyperparameters without validation.
In context
Questions about this term
What makes AdamW different from Adam?
The Decoupled Treatment of Weight Decay.
Is AdamW always better?
No, the choice depends on model, data, training goal and hyperparameters.
All details and background
In adaptive optimizers, classical L2 regularization is not generally equivalent to weight decay. AdamW therefore applies weight decay separately from the Adam update and makes regularization controllable separately.
ELI5 – Explained simply
Adam determines how learning updates are made; AdamW also reduces the weights in a separate step instead of mixing this effect into the same gradient.
Why it matters in practice
Relevant for own model training, if Weight Decay is to be controlled separately from adaptive gradient optimization.
Example
Possible example: When fine-tuning a Transformer model, a team uses AdamW and adjusts learning rate and weight decay separately using a validation set.
Pitfalls and limits
- Treat AdamW merely as a new name for Adam; equate weight decay and L2 regularization in adaptive optimizers; apply weight decay unchecked to all parameters; adopt hyperparameters without validation.
Origin & history
Ilya Loshchilov and Frank Hutter proposed Decoupled Weight Decay 2017; the work appeared at ICLR in 2019 and established the variant known as AdamW.
Use in mid-sized companies
- Possible applications: relevant for own model training and fine tuning, especially for Transformer-based models. Companies can thus make training recipes more reproducible, but must validate the effect on the basis of their own data.
Related terms
- Adam
- Weight Decay
- L2 regularisation
- Optimizers
- Transformers
- Fine tuning
Matching Flow-alerta service
AI opportunity analysis & roadmapAI potential analysis & roadmap: clarify tasks and data before a possible use of AI.Matching tool / use case
View use casesPyTorch AdamW; possible use: fine tuning with validated learning rate and weight decay.Sources
- arxiv.orgLoshchilov & Hutter (2017 / ICLR 2019), Decoupled Weight Decay Regularization
- docs.pytorch.orgPyTorch, AdamW
Quellenprüfung: 09.10.2026. Anwendungsbeispiele sind mögliche Szenarien, keine behaupteten Kundenfälle.
Editorially reviewed on: