LiteLLM
Definition
LiteLLM consolidates access to many model providers behind an OpenAI-compatible endpoint and adds routing, budgets, and protection rules.
LiteLLM is a self-hosted AI gateway. Applications send requests to a unified OpenAI-compatible endpoint; the gateway forwards them to configured model providers and can centrally manage access, costs, rate limits, fallbacks, logging, and guardrails.
Simply explained
Like a telephone switchboard, LiteLLM receives requests at one number and connects them according to rules to the appropriate model provider. Unlike a switchboard, the gateway does not harmonize the capabilities of connected models.
Why it matters in practice
A shared gateway layer reduces application-side differences between providers and creates a central control point. This facilitates provider switching, cost control, and operational policies, but also increases the gateway's importance for availability and security.
Example from business
Example (hypothetical): A company lets three internal applications use the same gateway and stores separate model approvals, budgets, and fallbacks per team.
Limits and pitfalls
- The gateway is a critical infrastructure component and must be secured, monitored, and operated with high availability. A unified API format does not mean models have identical capabilities, data policies, or response quality.
Origin & history
The project documentation describes LiteLLM as a self-hosted server with an OpenAI-compatible endpoint for more than one hundred LLM providers, including features for access, cost, reliability, guardrails, and logging.
Use in mid-sized companies
- Possible application in SMEs: Multiple AI applications can use provider access, budgets, logging, and fallback models via a shared controlled endpoint.
Frequently asked questions
What is LiteLLM?
A self-hosted gateway with an OpenAI-compatible endpoint for many model providers.
Does application code need to be changed when switching providers?
With compatible endpoints, the gateway can switch providers, but differences between models still need to be tested.
What controls does the gateway offer?
The documentation mentions virtual keys, budgets, rate limits, fallbacks, guardrails, and logging, among others.
Sources
- LiteLLM AI Gateway · BerriAI (o. J.): LiteLLM AI Gateway
Sources checked on 26.10.2026. Editorial use cases are hypothetical; no evidence of real customer results.
Editorially reviewed on: