Vanishing Gradients
Definition
Vanishing gradients weaken the learning signals in early parts of a neural network.
The strong decrease of backpropagated derivatives over many layers or time steps, causing early parameters to learn very little.
Simply explained
The feedback on the error becomes weaker as it passes through the network. Some parts therefore hardly learn anymore.
Why it matters in practice
The term helps to select, develop, evaluate AI systems in a well-founded manner and realistically assess their technical risks.
Example from business
A team identifies stagnating early layers and compares residual connections, normalization, and LSTM.
Limits and pitfalls
- Small gradients can also indicate saturation or implementation errors; countermeasures do not guarantee stable optimization.
Origin & history
Bengio, Simard, and Frasconi analyzed the problem of long-term dependencies in recurrent networks in 1994.
Use in mid-sized companies
- Possible example: A team identifies stagnating early layers and compares residual connections, normalization, and LSTM.
Frequently asked questions
What does vanishing gradients mean?
The strong decrease of backpropagated derivatives over many layers or time steps, causing early parameters to learn very little. What should be watched out for? Small gradients can also indicate saturation or implementation errors; countermeasures do not guarantee stable optimization.
Sources
- Learning Long-Term Dependencies · Bengio et al., Learning Long-Term Dependencies
Source checked on 10.10.2026. Editorial application examples are hypothetical; no proof of real customer results.
Editorially reviewed on: