<2-3 sentences summarising what the project addresses and solves, the target audience, and the benefit.> This project provides a unified proxy server for interacting with over 50 Large Language Models (LLMs) from various providers like OpenAI, Azure, Anthropic, and Hugging Face. It simplifies LLM integration for developers and researchers by offering a consistent OpenAI-compatible API, robust error handling with model fallbacks, and integrated logging, caching, and usage tracking features.

How It Works

The liteLLM proxy acts as an intermediary, accepting requests in the standard OpenAI /chat/completions format and routing them to the appropriate LLM backend. Its core advantage lies in abstracting away the complexities of different LLM APIs, providing a single interface. It enhances reliability through model fallback mechanisms, logs detailed request/response data to multiple platforms, tracks token usage and costs, and implements semantic caching for performance.

Quick Start & Requirements

• Primary install: Clone the repository (https://github.com/BerriAI/liteLLM-proxy) and install dependencies via pip install requirements.txt.
• Configuration: Set LLM API keys in a .env file or environment variables (e.g., OPENAI_API_KEY).
• Running: Start the proxy server with python3 main.py.
• Prerequisites: Python 3, API keys for desired LLM providers.
• Deployment: A Dockerfile is included for deployment on cloud platforms like Railway, GCP, AWS, and Azure.
• Documentation: Links for supported models and API inputs are available at https://docs.litellm.ai/docs/.

Highlighted Details

• Supports 50+ LLM models from diverse providers, with details at https://docs.litellm.ai/docs/.
• Consistent input/output format using the OpenAI API standard.
• Model fallback system for enhanced request reliability.
• Extensive logging integrations (Supabase, Posthog, Mixpanel, Sentry, etc.), with provider details at https://docs.litellm.ai/docs/.
• Token usage and spend tracking per model.
• Implementation of semantic caching.
• Support for streaming responses and asynchronous operations.

Maintenance & Community

• Community support is available via Discord.
• Contact information for founders is provided: ishaan@berri.ai, krrish@berri.ai.
• A roadmap outlines planned features including hosted database support, enhanced logging destinations, caching improvements, rate limiting, and spending controls.

Licensing & Compatibility

• The repository's license is not explicitly stated in the provided README.
• Compatibility for commercial use or closed-source linking is undetermined without a specified license.

Limitations & Caveats

• DEPRECATION WARNING: This project is deprecated. Development and support have moved to a new home, LiteLLM. Users are strongly advised to migrate to the new LiteLLM project.