Skip to content
    Flow-alerta.cab

    Throughput

    Web design & developmentSystem flow rateProcessing rate
    Editorially reviewed on: 1 source
    Your reading view

    The short answer · 16 words

    In short

    Measuring the amount of data or transaction that a system processes within a given time unit.

    Flow-alerta.cab (2026): Throughput. https://flow-alerta.cab/en/glossar/throughput. Editorially reviewed on: 09 October 2026.

    How much detail would you like?

    Practical

    Relevant for capacity planning and load testing; together with latency, error rate and cost measurement.

    Possible example: A team compares the total token throughput of two server configurations at the same test load. From this, a proportional increase of simultaneous chats is not automatically derived.

    Dimensioning of cloud servers, load tests for customer portals and cost optimization in AI hosting.

    At a glance

    Core
    Throughput measures processed work per unit of time, such as requests per second or generated tokens per second. In the LLM comparison, it must be clear whether input or output tokens and the overall system or a single request are measured. High throughput does not guarantee low response time or low overall costs.
    Use
    Relevant for capacity planning and load testing; together with latency, error rate and cost measurement.
    Example
    Possible example: A team compares the total token throughput of two server configurations at the same test load. From this, a proportional increase of simultaneous chats is not automatically derived.
    Limits
    Confuse overall throughput with speed per user; compare different token lengths and hardware; ignore queues, errors and downstream bottlenecks.

    Questions about this term

    What is the difference with latency?

    Throughput measures amount per time; latency the duration of a single processing. For chats, time to the first token and speed of further output are also important.

    How can throughput be improved?

    For example, by batching or scaling; effect, response time and cost must be measured under comparable load.

    All details and background

    Throughput measures processed work per unit of time, such as requests per second or generated tokens per second. In the LLM comparison, it must be clear whether input or output tokens and the overall system or a single request are measured. High throughput does not guarantee low response time or low overall costs.

    ELI5 – Explained simply

    Like the number of vehicles that pass a toll point per hour. This does not say how long a single vehicle has to wait.

    Why it matters in practice

    Relevant for capacity planning and load testing; together with latency, error rate and cost measurement.

    Example

    Possible example: A team compares the total token throughput of two server configurations at the same test load. From this, a proportional increase of simultaneous chats is not automatically derived.

    Pitfalls and limits

    • Confuse overall throughput with speed per user; compare different token lengths and hardware; ignore queues, errors and downstream bottlenecks.

    Origin & history

    Classical concept of computer architecture and network technology, refocused in the AI sector by LLM inference engines such as vLLM.

    Use in mid-sized companies

    • Dimensioning of cloud servers, load tests for customer portals and cost optimization in AI hosting.

    Related terms

    • Latency
    • TPS (Tokens per Second)
    • vLLM
    • Load distribution
    • Scalability

    Matching Flow-alerta service

    Websites & web applicationsWebsites & Web Applications; AI Assistant & Automation – Checking Capacity and Response Times

    Matching tool / use case

    View use casesUse Case: Check a web API or AI server under documented load for throughput, latency and errors.

    Sources

    • arxiv.orgKwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention

    Quellen geprüft am 09.10.2026.

    Editorially reviewed on:

    Next step

    Put knowledge into practice

    Discuss your projectView use cases