AI Proxy is a production-ready, high-performance AI gateway designed to act as a central middleware for AI applications. It offers an OpenAI-compatible protocol endpoint, supporting models from OpenAI, Anthropic, and Gemini, while providing intelligent request routing, comprehensive monitoring, and robust multi-tenant management. The gateway aims to enhance reliability, scalability, and manageability for AI-driven services.

How It Works

The core of AI Proxy is its intelligent request management system, featuring smart retry logic, priority-based channel selection, and load balancing across various AI providers. It excels at protocol conversion, seamlessly translating between OpenAI Chat Completions, Claude Messages, Gemini, and OpenAI Responses API formats, enabling unified access to diverse models. Its multi-tenant architecture enforces organization isolation, flexible access control, and resource quotas, making it suitable for shared environments. An extensible plugin system further enhances functionality with features like caching and web search.

Quick Start & Requirements

Installation is recommended via Docker or Docker Compose, with a docker run command and a docker-compose.yaml file provided. Building from source requires Go 1.24+ and Node.js 22+ (for frontend development). Optional dependencies include PostgreSQL for the primary database and Redis for caching. An interactive API explorer is available at http://localhost:3000/swagger/index.html.

Highlighted Details

• Supports multiple AI providers (OpenAI, Anthropic, Azure OpenAI, Gemini) via an OpenAI-compatible API.
• Features an extensible plugin system including Cache, Web Search, Think Split, and Stream Fake plugins.
• Implements Model Context Protocol (MCP) support, enabling automatic tool generation from OpenAPI specs.
• Provides a management panel for configuration and monitoring, with detailed analytics on request volume, errors, and costs.
• Offers robust multi-tenant capabilities with organization isolation, token-based authentication, and resource quotas (RPM/TPM).

Maintenance & Community

The project welcomes contributions via bug reports, feature suggestions, documentation improvements, and pull requests. It acknowledges OpenAI and the broader open-source community. Specific community channels (e.g., Discord, Slack) or a public roadmap are not detailed in the README.

Licensing & Compatibility

AI Proxy is released under the MIT License, which is permissive and generally compatible with commercial use and linking in closed-source applications.

Limitations & Caveats

The README does not explicitly list limitations, alpha/beta status, or known bugs. Building from source has specific Go and Node.js version requirements. The default setup uses SQLite, with PostgreSQL and Redis noted as optional enhancements for production.