Layer Normalization
Definition
Layer normalization normalizes activations within a single example across defined feature dimensions.
Unlike batch normalization, LayerNorm does not require batch statistics and is therefore well suited for sequence models and transformers.
Simply explained
How to bring values within a single data point to a common scale.
Why it matters in practice
Central for transformers and many modern neural networks.
Example from business
A transformer block uses LayerNorm before or after attention and feedforward layers.
Limits and pitfalls
- Equate LayerNorm with BatchNorm; choose normalization axes incorrectly; exchange architectural variants without verification.
Origin & history
Ba, Kiros, and Hinton introduced Layer Normalization in 2016.
Use in mid-sized companies
- Possible application in SMEs: Understand architectures and implement reproducibly.
Frequently asked questions
Is LayerNorm batch-dependent?
No. Is it always before attention? That depends on the architecture.
Sources
- Layer Normalization · Ba et al. (2016), Layer Normalization
Sources checked on 10.10.2026.
Editorially reviewed on: