Leaky ReLU
Definition
Leaky ReLU is an activation function that uses a small linear slope for negative inputs instead of exactly zero.
It aims to reduce the problem of permanently inactive ReLU units. The negative slope parameter is a model decision.
Simply explained
Like a ReLU door that doesn’t fully close in the negative range.
Why it matters in practice
Leaky ReLU is a variant for activating neural networks. Its classification helps with understanding and tuning such architectures.
Example from business
An image model compares ReLU and Leaky ReLU with otherwise identical architecture.
Limits and pitfalls
- Treat Leaky ReLU as generally better; adopt slope unchecked.
Origin & history
Maas, Hannun, and Ng studied non-saturating activations with a small negative gradient in 2013.
Use in mid-sized companies
- Possible application in SMEs: Test as an architectural variant in controlled model comparisons.
Frequently asked questions
What is the difference to ReLU?
Negative values are not completely set to zero.
Sources
- Rectifier Nonlinearities Improve Neural Network Acoustic Models · Maas et al. (2013), Rectifier Nonlinearities Improve Neural Network Acoustic Models
Sources checked on 10.10.2026.
Editorially reviewed on: