Quadratic Attention Cost
Definition
Full attention is costly for long inputs.
With full self-attention, relationships between token pairs are calculated; their number grows quadratically with sequence length.
Simply explained
If every child is to talk to every other, the number of conversation pairs grows quickly.
Why it matters in practice
Longer contexts can cause significantly more computation.
Example from business
A hypothetical team compares short excerpts with full documents in a model query.
Limits and pitfalls
- Optimized implementations change memory requirements and runtime; not every architecture uses full attention.
Origin & history
This relationship is part of the fundamentals of the Transformer architecture.
Use in mid-sized companies
- Limit context to relevant content and measure actual runtime.
Frequently asked questions
Does duplicated text always double the total cost?
No, it depends on the model and implementation.
Sources
Editorially reviewed on: