Pre-LayerNorm and Post-LayerNorm
Definition
Pre-Norm and Post-Norm distinguish the position of normalization in the Transformer.
Two Transformer configurations where Layer Normalization is placed before or after attention or feed-forward sub-blocks.
Simply explained
The values are adjusted either before or after a computational block.
Why it matters in practice
The term helps to select, develop, evaluate AI systems in a well-founded manner and realistically assess their technical risks.
Example from business
An architecture compares training stability and quality of both configurations.
Limits and pitfalls
- Portraying Pre-LN as generally better or ignoring differences in depth, initialization, and residual path.
Origin & history
The original Transformer used Post-LN; Pre-LN became widespread due to often more stable training.
Use in mid-sized companies
- Possible application: An architecture compares training stability and quality of both configurations.
Frequently asked questions
What do Pre-LayerNorm and Post-LayerNorm mean?
Two Transformer configurations where Layer Normalization is placed before or after attention or feed-forward sub-blocks. What mistakes should be avoided? Portraying Pre-LN as generally better or ignoring differences in depth, initialization, and residual path.
Sources
Source checked on 10.10.2026. Editorial application examples are hypothetical; no proof of real customer results.
Editorially reviewed on: