Stochastic Weight Averaging (SWA)
Definition
SWA averages model weights from several later training states to obtain a potentially better transferable solution.
SWA computes the arithmetic mean of model weights from multiple later training states, often along a learning rate trajectory, which can result in a better generalizing parameter solution.
Simply explained
Instead of keeping only the last training state, an average is formed from several late states. This can help the model with new data.
Why it matters in practice
The term helps to select, develop, evaluate AI systems in a well-founded manner and realistically assess their technical risks.
Example from business
A team collects weights from the last training epochs with cyclical learning rate and evaluates the averaged model.
Limits and pitfalls
- Treating wider or flatter loss regions as coordinate-independent guarantees, averaging weights from incompatible training runs, or not updating batch normalization statistics after averaging.
Origin & history
Izmailov and co-authors published SWA in 2018 as a simple method to improve generalization.
Use in mid-sized companies
- Possible example: A team collects weights from the last training epochs with cyclical learning rate and evaluates the averaged model.
Frequently asked questions
What does Stochastic Weight Averaging (SWA) mean?
SWA computes the arithmetic mean of model weights from multiple later training states, often along a learning rate trajectory, which can result in a better generalizing parameter solution. What mistakes should be avoided? Treating wider or flatter loss regions as coordinate-independent guarantees, averaging weights from incompatible training runs, or not updating batch normalization statistics after averaging.
Sources
- Averaging Weights Leads to Wider Optima and Better Generalization · Izmailov et al., Averaging Weights Leads to Wider Optima and Better Generalization
Source checked on 10.10.2026. Editorial application examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: