Linear Attention
Definition
Linear attention refers to attention variants whose computational or memory complexity scales approximately linearly rather than quadratically with sequence length.
Methods use kernel tricks, factorizations, or other transformations to avoid the full attention matrix. Quality and actual hardware efficiency vary greatly.
Simply explained
How to process long sequences without comparing every token directly with every other.
Why it matters in practice
Relevant for efficient long sequences, but no single algorithm.
Example from business
A research team compares standard attention and linear variant on long sensor data.
Limits and pitfalls
- Misinterpret "linear" as always faster; ignore model quality and hardware kernels; equate different methods.
Origin & history
Katharopoulos et al. demonstrated in 2020 Transformer formulations with linear attention and recurrent view.
Use in mid-sized companies
- Possible application in SMEs: Use only in specialized models after real quality/latency benchmarking.
Frequently asked questions
Is Linear Attention a concrete method?
It is a family. Does it generally replace Softmax Attention? No.
Sources
- Transformers are RNNs · Katharopoulos et al. (2020), Transformers are RNNs
Sources checked on 10.10.2026.
Editorially reviewed on: