Sliding Window Attention
Definition
Attention with sliding window limits the considered context to a local environment.
Attention with sliding window restricts each token to a local context area to reduce computational and memory costs of long sequences.
Simply explained
When reading, the model looks at only a limited section near each position.
Why it matters in practice
The term helps to select, develop, evaluate AI systems in a well-founded manner and realistically assess their technical risks.
Example from business
A document model processes long contracts by having each token primarily consider nearby text sections.
Limits and pitfalls
- Local windows can miss distant dependencies; window size and global connections are crucial.
Origin & history
Longformer described in 2020 local windows combined with global attention; later models use related variants.
Use in mid-sized companies
- Possible application: A document model processes long contracts by having each token primarily consider nearby text sections.
Frequently asked questions
What does Attention with sliding window mean?
Attention with sliding window restricts each token to a local context area to reduce computational and memory costs of long sequences. What should be considered? Local windows can miss distant dependencies; window size and global connections are crucial.
Sources
- Longformer: The Long-Document Transformer · Beltagy et al., Longformer: The Long-Document Transformer
Source checked on 10.10.2026. Editorial application examples are hypothetical; no evidence of real customer results.
Editorially reviewed on: