Skip to content
    Flow-alerta.cab

    Gradient Clipping

    AI fundamentals & modelsGradient limitationGradient norm clipping
    Editorially reviewed on: 2 sources
    Your reading view

    The short answer · 10 words

    In short

    Gradient clipping limits to large gradients before the optimizer step.

    Flow-alerta.cab (2026): Gradient Clipping. https://flow-alerta.cab/en/glossar/gradient-clipping. Editorially reviewed on: 10 October 2026.

    How much detail would you like?

    Practical

    Clipping can stabilize training, but does not replace a root cause analysis of learning rate, data or architecture.

    Hypothetical example: A sequence model logs gradient norms and clips above a validated limit.

    Select the limit based on your training diagnostics and examine frequent clipping and NaN levels.

    At a glance

    Core
    In standard clipping, the entire gradient is scaled when its norm exceeds a limit; value clipping limits individual values. It treats symptoms of unstable updates.
    Use
    Clipping can stabilize training, but does not replace a root cause analysis of learning rate, data or architecture.
    Example
    Hypothetical example: A sequence model logs gradient norms and clips above a validated limit.
    Limits
    Adopt limit blindly; hide permanent clipping; mix value and norm clipping; do not examine NaNs.

    In context

    Backpropagation

    Efficient method for calculating gradients in multilayer neural networks.

    Questions about this term

    Does it fix vanishing gradients?

    No.

    When does it work?

    If the defined limit value is exceeded.

    All details and background

    In standard clipping, the entire gradient is scaled when its norm exceeds a limit; value clipping limits individual values. It treats symptoms of unstable updates.

    ELI5 – Explained simply

    Too large gradients are limited before the parameter step; the actual step size additionally depends on the optimizer.

    Why it matters in practice

    Clipping can stabilize training, but does not replace a root cause analysis of learning rate, data or architecture.

    Example

    Hypothetical example: A sequence model logs gradient norms and clips above a validated limit.

    Pitfalls and limits

    • Adopt limit blindly; hide permanent clipping; mix value and norm clipping; do not examine NaNs.

    Origin & history

    Pascanu, Mikolov and Bengio described standard clipping against exploding gradients in the preprint of 2012, revised 2013.

    Use in mid-sized companies

    • Select the limit based on your training diagnostics and examine frequent clipping and NaN levels.

    Related terms

    Matching Flow-alerta service

    AI opportunity analysis & roadmapAI consulting: Examine tasks, data and prerequisites of a possible AI use case.

    Matching tool / use case

    View use casesPyTorch torch.nn.utils.clip_grad_norm_; logging of the gradient norm and non-finite values.

    Sources

    Quellenprüfung: 10.10.2026. Definition und dokumentierte Einordnung abgeglichen; Beispiele beschreiben mögliche Anwendungen.

    Editorially reviewed on:

    Next step

    Put knowledge into practice

    Discuss your projectView use cases