Causal Masking
The short answer · 17 words
In short
Attention mask that prevents a model from accessing future positions of the sequence when predicting a token.
Flow-alerta.cab (2026): Causal Masking. https://flow-alerta.cab/en/glossar/causal-masking. Editorially reviewed on: 09 October 2026.
How much detail would you like?
Practical
Basic mechanism of autoregressive language models and important for understanding training, decoding and data leakage within a sequence.
Example scenario: When training a text generator, access to the tokens is masked for position 5 at position 6 and later.
In the case of own autoregressive models, separately test masking, padding and position logic; in the case of finished APIs, this is usually an internal architectural mechanism.
At a glance
- Core
- In the case of causal masking, a position may take into account only itself and previous positions. Transformer decoders block the attention to later tokens. Together with correctly shifted inputs by one position, this prevents later outputs from being used in the autoregressive prediction.
- Use
- Basic mechanism of autoregressive language models and important for understanding training, decoding and data leakage within a sequence.
- Example
- Example scenario: When training a text generator, access to the tokens is masked for position 5 at position 6 and later.
- Limits
- Confuse causal mask with padding mask; let future tokens be visible in training; assume any Transformer architecture uses causal masking.
In context
Questions about this term
Is causal masking the same as padding masking?
No.
Why is it masked?
So that future tokens do not flow into the current prediction.
Do encoders like BERT use the same mask?
Not in classic bidirectional BERT.
All details and background
In the case of causal masking, a position may take into account only itself and previous positions. Transformer decoders block the attention to later tokens. Together with correctly shifted inputs by one position, this prevents later outputs from being used in the autoregressive prediction.
ELI5 – Explained simply
If the model is to predict the next word, it must not secretly see the later words.
Why it matters in practice
Basic mechanism of autoregressive language models and important for understanding training, decoding and data leakage within a sequence.
Example
Example scenario: When training a text generator, access to the tokens is masked for position 5 at position 6 and later.
Pitfalls and limits
- Confuse causal mask with padding mask; let future tokens be visible in training; assume any Transformer architecture uses causal masking.
Origin & history
The 2017 Transformer paper describes in the decoder the masking of subsequent positions, so that predictions depend only on already known outputs.
Use in mid-sized companies
- In the case of own autoregressive models, separately test masking, padding and position logic; in the case of finished APIs, this is usually an internal architectural mechanism.
Related terms
- Self-attention
- Autoregressive model
- Transformers
- Decoders
- Tokens
Matching Flow-alerta service
AI adoption & supportAI implementation & support: clarify usage guidelines, limits, responsibilities and escalation paths.Matching tool / use case
View use casesUse Case: Autoregressive training of a Transformer decoder without access to future tokens.Sources
- arxiv.orgVaswani et al., Attention Is All You Need (2017), Abschnitt 3.1 und 3.2.3
Quellenprüfung: 09.10.2026. Definition und dokumentierte Einordnung abgeglichen; Beispiele beschreiben mögliche Anwendungen.
Editorially reviewed on: