litellm
by BerriAIPython SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, VLLM, NVIDIA NIM]
About litellm
litellm is a Python open-source devops & mlops project by BerriAI. Python SDK, Proxy Server (AI Gateway) to call 100+ LLM APIs in OpenAI (or native) format, with cost tracking, guardrails, loadbalancing and logging. [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, VLLM, NVIDIA NIM] With 51,145 GitHub stars and 9,072forks, it's one of the DevOps & MLOps tools worth knowing. You can download the source, browse it on GitHub, or read the full setup guide in the README below.
litellm — guide
One interface over many providers
Every model vendor ships its own SDK, its own request shape and its own error semantics. LiteLLM flattens that: you write code once against an OpenAI-shaped interface, and it translates to whichever provider you name — Anthropic, Azure, Bedrock, Vertex, Cohere and a long tail of others. Changing model becomes a string change rather than a rewrite, which is the entire point when you are still deciding what to run in production.
The SDK and the proxy are different products
This trips people up. The Python SDK is a library you import into one application. The proxy server is a gateway you deploy once, in front of everything — and that is where the operational features live: per-key spend limits and cost tracking, load balancing across deployments, automatic fallback when a provider errors or rate-limits, guardrails, and a single log of every call your organisation makes. If the problem is "which model do I call", use the SDK. If it is "what is my team spending and who is calling what", you want the proxy.
What it costs you
An extra hop. Self-hosting the proxy means one more service to run and monitor, and it sits on the critical path of every model call — worth sizing and watching accordingly. Provider-specific features sometimes need the native request format rather than the translated one, which the library supports but which gives back a little of the portability you adopted it for.
When to pick something else
Calling exactly one provider forever? Use their SDK directly and skip the layer. LiteLLM earns its keep the moment you have two providers, or need to answer a question about spend.
From the project README
🚅 LiteLLM
LiteLLM AI Gateway
Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.
This is an excerpt from the project's own README, quoted here for reference. Read the full README on GitHub →
litellm — FAQ
What is the difference between the LiteLLM SDK and the proxy?
The SDK is a library inside one application that normalises provider APIs. The proxy is a gateway you deploy in front of everything, and it is where cost tracking, spend limits, load balancing, fallbacks and centralised logging live. Teams generally want the proxy.
Does LiteLLM add latency?
The SDK adds almost nothing — it is a translation layer in-process. The self-hosted proxy adds a real network hop, so it sits on the critical path of every call and should be sized and monitored like any other production service.
Can LiteLLM fall back to another provider automatically?
Yes, through the proxy. You configure an ordered list of deployments and it retries against the next one when a provider errors or rate-limits, which is one of the main reasons to run the gateway rather than the library alone.
Is LiteLLM free?
The open-source SDK and proxy are free and self-hostable under an MIT licence. The company sells an enterprise tier with additional administrative features, but the core gateway is fully usable without it.