Prefill
Definition
Prefill processes the input context before the start of stepwise text generation.
The inference phase in which a model processes all input tokens and builds the initial KV cache before autoregressive output begins.
Simply explained
First, the model works through the provided information and prepares it for its response.
Why it matters in practice
The term helps to select, develop, evaluate AI systems in a well-founded manner and realistically assess their technical risks.
Example from business
A long system context causes high Time to First Token, although individual output tokens follow quickly.
Limits and pitfalls
- Processing the input is only part of the response time. Its duration must not be confused with total latency or text generation speed.
Origin & history
With large language models, the separation of Prefill and Decode became important for performance planning.
Use in mid-sized companies
- Possible application: A long system context causes high Time to First Token, although individual output tokens follow quickly.
Frequently asked questions
What does processing the input context mean?
The inference phase in which a model processes all input tokens and builds the initial KV cache before autoregressive output begins. What mistakes should be avoided? Confusing prefill time with total latency or decode throughput.
Sources
Source checked on 10.10.2026. Editorial application examples are hypothetical; no proof of real customer results.
Editorially reviewed on: