KV-Cache
Definition
A KV cache stores already computed key and value representations of an autoregressive transformer model for later tokens.
During generation, previous attention states do not need to be fully recomputed for each new token. This reduces computation but increases memory usage.
Simply explained
How to store intermediate results so that a long text does not have to be recomputed from scratch for every new word.
Why it matters in practice
Relevant for LLM latency, throughput, and memory planning.
Example from business
An inference server measures how much longer contexts occupy GPU memory through KV caches.
Limits and pitfalls
- Equating KV cache with context window; underestimating memory requirements; cache sharing without privacy checks.
Origin & history
KV caching is a practical optimization for autoregressive transformer inference and is widely used in modern inference libraries.
Use in mid-sized companies
- Possible application in SMEs: Plan memory, batch size, and context length jointly for LLM serving.
Frequently asked questions
Does KV cache speed up training?
It is primarily used for autoregressive inference. Does it consume memory? Yes.
Sources
- Cache strategies · Hugging Face Transformers, Cache strategies
Sources checked on 10.10.2026.
Editorially reviewed on: