Skip to content
    Flow-alerta.cab

    AdaGrad

    AI fundamentals & modelsAdaGradeAdaptive Gradient AlgorithmAdaptive subgradient method
    Editorially reviewed on: 2 sources
    Your reading view

    The short answer · 19 words

    In short

    AdaGrad is an adaptive gradient method that automatically adjusts the learning rate for individual parameters based on earlier gradients.

    Flow-alerta.cab (2026): AdaGrad. https://flow-alerta.cab/en/glossar/adagrad. Editorially reviewed on: 09 October 2026.

    How much detail would you like?

    Practical

    Relevant for adaptive optimization, especially for sparsely populated features; suitability depends on the training problem.

    Possible example: In a model with many rare text features, AdaGrad can update rare but informative features with relatively larger effective learning steps.

    Possible applications: relevant for in-house ML training, especially for sparse features, online learning or models with very different frequently updated parameters. For pure API use of finished AI models, AdaGrad is hardly operationally relevant.

    At a glance

    Core
    In the common diagonal variant, AdaGrad accumulates squared gradients per parameter. Larger summed gradients reduce its effective step size.
    Use
    Relevant for adaptive optimization, especially for sparsely populated features; suitability depends on the training problem.
    Example
    Possible example: In a model with many rare text features, AdaGrad can update rare but informative features with relatively larger effective learning steps.
    Limits
    The learning rate can shrink too much due to the continuous accumulation of squared gradients; still choose global learning rate incorrectly; consider AdaGrad as the best choice for deep networks; perform optimizer comparisons without identical training conditions.

    In context

    Adam Optimizer

    Adam is an optimization algorithm for training neural networks that derives adaptive learning rates from moment estimates of the gradients.

    AdamW

    AdamW is a variant of Adam in which Weight Decay is decoupled from the gradient-based optimization step.

    Adafactor

    Adafactor is a memory-efficient adaptive optimization algorithm for training large neural networks.

    Questions about this term

    Why is AdaGrad called adaptive?

    Because effective step sizes are derived from previous gradients.

    What is a weakness?

    The accumulated gradients can make the effective learning rate very small in the long term.

    All details and background

    In the common diagonal variant, AdaGrad accumulates squared gradients per parameter. Larger summed gradients reduce its effective step size.

    ELI5 – Explained simply

    Each parameter gets its own tempo control: what has often been changed is adjusted more cautiously; rare signals can get larger steps.

    Why it matters in practice

    Relevant for adaptive optimization, especially for sparsely populated features; suitability depends on the training problem.

    Example

    Possible example: In a model with many rare text features, AdaGrad can update rare but informative features with relatively larger effective learning steps.

    Pitfalls and limits

    • The learning rate can shrink too much due to the continuous accumulation of squared gradients; still choose global learning rate incorrectly; consider AdaGrad as the best choice for deep networks; perform optimizer comparisons without identical training conditions.

    Origin & history

    In 2011, John Duchi, Elad Hazan and Yoram Singer published in JMLR a family of adaptive subgradient methods that uses information from previous gradients to dynamically adjust optimization.

    Use in mid-sized companies

    • Possible applications: relevant for in-house ML training, especially for sparse features, online learning or models with very different frequently updated parameters. For pure API use of finished AI models, AdaGrad is hardly operationally relevant.

    Related terms

    Matching Flow-alerta service

    AI opportunity analysis & roadmapAI potential analysis & roadmap: clarify tasks and data before a possible use of AI.

    Matching tool / use case

    View use casesPyTorch Adagrad; possible use: training with sparsely populated features.

    Sources

    Quellenprüfung: 09.10.2026. Anwendungsbeispiele sind mögliche Szenarien, keine behaupteten Kundenfälle.

    Editorially reviewed on:

    Next step

    Put knowledge into practice

    Discuss your projectView use cases