Group Normalization
Definition
Group Normalization normalizes activations within channel groups instead of across the batch dimension.
The method divides feature channels into groups and computes mean and variance within each group. Thus, normalization does not depend on batch size.
Simply explained
How to split multiple measurement channels into small groups and bring each group to a comparable scale.
Why it matters in practice
Relevant for deep learning models with small batches, for example when memory requirements prevent large batch sizes.
Example from business
An image model is trained with small batches; GroupNorm replaces BatchNorm to depend less on fluctuating batch statistics.
Limits and pitfalls
- Choosing too many or inappropriate groups; presenting GroupNorm as generally better than BatchNorm; not measuring architecture and data effects.
Origin & history
Wu and He introduced Group Normalization in 2018 as an alternative to Batch Normalization, especially for small batch sizes.
Use in mid-sized companies
- Possible application: Training or fine-tuning specialized computer vision models with limited GPU memory.
Frequently asked questions
Is GroupNorm independent of batch size?
The calculation does not use batch statistics. Is it always better than BatchNorm? No, it depends on the model, data, and training regime.
Sources
- Group Normalization · Wu & He (2018), Group Normalization
Sources checked on 10.10.2026.
Editorially reviewed on: