Adafactor
The short answer · 12 words
In short
Adafactor is a memory-efficient adaptive optimization algorithm for training large neural networks.
Flow-alerta.cab (2026): Adafactor. https://flow-alerta.cab/en/glossar/adafactor. Editorially reviewed on: 09 October 2026.
How much detail would you like?
Practical
Relevant when training very large models, when optimizer states occupy a lot of memory. For ordinary AI applications that use finished models via API, however, Adafactor usually does not play a direct role.
Possible example: When training a large transformer model, Adafactor reduces memory for estimating second moments, allowing more model parameters or larger batches to fit into the available accelerator memory.
Possible applications: especially relevant for companies that have to train or fine-tune models themselves and optimize GPU/TPU memory and training costs. For typical medium-sized projects with API models, the benefit is rather indirect.
At a glance
- Core
- Adafactor scales parameter updates based on estimated second gradient moments. In the case of matrix parameters, it factors this estimate over rows and columns, thereby reducing the additional memory requirement compared to optimizers with complete moment estimates.
- Use
- Relevant when training very large models, when optimizer states occupy a lot of memory. For ordinary AI applications that use finished models via API, however, Adafactor usually does not play a direct role.
- Example
- Possible example: When training a large transformer model, Adafactor reduces memory for estimating second moments, allowing more model parameters or larger batches to fit into the available accelerator memory.
- Limits
- Equate memory advantage with automatically better model quality; adopt hyperparameters unchecked by Adam; ignore relative increments and update clipping; consider the optimizer as a substitute for clean training and evaluation procedures.
In context
Adam Optimizer
Adam is an optimization algorithm for training neural networks that derives adaptive learning rates from moment estimates of the gradients.
AdamW
AdamW is a variant of Adam in which Weight Decay is decoupled from the gradient-based optimization step.
AdaGrad
AdaGrad is an adaptive gradient method that automatically adjusts the learning rate for individual parameters based on earlier gradients.
Questions about this term
Is Adafactor simply Adam with less memory?
No, the procedures differ, among other things, in the moment estimation and proposed update rules.
Is Adafactor relevant for inference?
Primarily for training or Fine tuning.
All details and background
Adafactor scales parameter updates based on estimated second gradient moments. In the case of matrix parameters, it factors this estimate over rows and columns, thereby reducing the additional memory requirement compared to optimizers with complete moment estimates.
ELI5 – Explained simply
Like a space-saving notepad: For a large number matrix, statistics are stored over rows and columns instead of logging each number individually.
Why it matters in practice
Relevant when training very large models, when optimizer states occupy a lot of memory. For ordinary AI applications that use finished models via API, however, Adafactor usually does not play a direct role.
Example
Possible example: When training a large transformer model, Adafactor reduces memory for estimating second moments, allowing more model parameters or larger batches to fit into the available accelerator memory.
Pitfalls and limits
- Equate memory advantage with automatically better model quality; adopt hyperparameters unchecked by Adam; ignore relative increments and update clipping; consider the optimizer as a substitute for clean training and evaluation procedures.
Origin & history
Noam Shazeer and Mitchell Stern introduced Adafactor in 2018 in “Adafactor: Adaptive Learning Rates with Sublinear Memory Cost”. In particular, the aim was to reduce the memory requirement of adaptive optimisation in large weight matrices.
Use in mid-sized companies
- Possible applications: especially relevant for companies that have to train or fine-tune models themselves and optimize GPU/TPU memory and training costs. For typical medium-sized projects with API models, the benefit is rather indirect.
Related terms
- Adam
- AdamWW
- AdaGrade
- Optimizers
- Gradient descent
- Transformers
- Fine tuning
Matching Flow-alerta service
AI opportunity analysis & roadmapAI potential analysis & roadmap: clarify tasks and data before a possible use of AI.Matching tool / use case
View use casesPyTorch Adafactor; possible use: memory-efficient training or fine-tuning of large weight matrices.Sources
- arxiv.orgShazeer & Stern (2018), Adafactor
- docs.pytorch.orgPyTorch, Adafactor
Quellenprüfung: 09.10.2026. Anwendungsbeispiele sind mögliche Szenarien, keine behaupteten Kundenfälle.
Editorially reviewed on: