LLM Routing
Definition
LLM Routing selects a suitable model or backend depending on task, cost, latency, or risk.
Routing can be rule-based or learning-based and differentiate between providers, model sizes, or specialized models. Fallbacks and evaluation data are crucial.
Simply explained
How to forward requests to the appropriate specialist depending on difficulty.
Why it matters in practice
Relevant for cost control and robust multi-model systems.
Example from business
Simple extractions go to a small model, difficult analyses to a more powerful model.
Limits and pitfalls
- Routing only by price; not evaluating model capabilities; fallback loops; ignoring different data conditions.
Origin & history
Model Routing emerged as a practical pattern with multi-provider and multi-model infrastructures and is now implemented in AI gateways and serving platforms.
Use in mid-sized companies
- Possible application in SMEs: define task classes and evaluate routing against quality, cost, and latency.
Frequently asked questions
Is routing automatically intelligent?
No. Can a gateway handle routing? Often yes.
Sources
- AI Gateway · Microsoft Azure API Management, AI Gateway
- Ergänzender Primärbeleg · Ergänzender Primärbeleg
Sources verified: 10./12.10.2026.
Editorially reviewed on: