LAMB Optimizer
Definition
LAMB is an optimizer for training with very large batches that combines Adam-like updates with layer-wise scaling.
The method was developed for large-batch training of large models. Later work showed its advantages are not universal and strong standard optimizers can compete.
Simply explained
How to complement Adam with a controller per layer to better handle very large batch sizes.
Why it matters in practice
Relevant mainly for large-scale model training, not for normal AI usage.
Example from business
A research team compares LAMB and AdamW in pretraining under identical compute budget.
Limits and pitfalls
- Presenting LAMB as generally superior; transferring results from BERT pretraining to other workloads; increasing batch size without generalization test.
Origin & history
You et al. published LAMB in 2019 for large-batch training of BERT; later reality checks relativized universal advantages.
Use in mid-sized companies
- Possible application in SMEs: Check only with own large-scale training and reproducible benchmarks.
Frequently asked questions
Is LAMB for inference?
No. Do you need it for fine-tuning small models? Usually not.
Sources
- Large Batch Optimization for Deep Learning · You et al. (2019), Large Batch Optimization for Deep Learning
Sources checked on 10.10.2026.
Editorially reviewed on: