Latency Budget
Definition
A latency budget allocates a maximum acceptable response time across multiple components of a system.
Instead of measuring only end-to-end latency, network, database, model, tools, or rendering receive their own budgets. This is an architectural practice, not a universal standard.
Simply explained
How to divide an overall deadline among multiple work steps.
Why it matters in practice
Relevant for voice, agents, APIs, and websites where individual slow components dominate the user experience.
Example from business
For an assistant, 2 seconds total target is distributed across retrieval, model, tool calls, and rendering.
Limits and pitfalls
- Only budgeting average latency; ignoring tail latency; setting unrealistic budgets without load testing.
Origin & history
Latency budgets are derived from performance engineering and SRE practice; distributed systems pay special attention to tail latency.
Use in mid-sized companies
- Possible application in SMEs: planning time-critical workflows with p95/p99 targets per component.
Frequently asked questions
Is a latency budget an SLA?
Not necessarily; it can be an internal engineering goal.
Sources
- The Tail at Scale · Dean & Barroso (2013), The Tail at Scale
- Ish-Shalom, Latenzbudget, Folie 24 · USENIX SREcon 2021, Ish-Shalom, Latenzbudget, Folie 24
Sources checked: 10./12.10.2026.
Editorially reviewed on: