AI Gateway provides a unified, high-performance interface to various Large Language Models (LLMs), abstracting away provider-specific APIs into a single OpenAI-compatible format. It is designed for developers and enterprises seeking to manage, secure, and optimize AI traffic, offering features like analytics, cost control, and advanced routing.

How It Works

Built in Rust, AI Gateway prioritizes speed and reliability. It acts as a reverse proxy, accepting requests in the OpenAI API format and forwarding them to configured LLM providers (OpenAI, Gemini, Anthropic, etc.). Its architecture supports advanced features like dynamic routing (fallback, script-based, latency-based), rate limiting, and cost controls, all configurable via YAML or command-line arguments. Observability is achieved through OpenTelemetry tracing, with ClickHouse as a backend for storing detailed usage analytics and request traces.

Quick Start & Requirements

• Installation:
  • Docker: docker run -it -p 8080:8080 -e LANGDB_KEY=your-langdb-key-here langdb/ai-gateway serve
  • Cargo: export RUSTFLAGS="--cfg tracing_unstable --cfg aws_sdk_unstable" cargo install ai-gateway; export LANGDB_KEY=your-langdb-key-here; ai-gateway serve
• Prerequisites: Rust toolchain for source install. Docker for containerized deployment. API keys for LLM providers. ClickHouse is recommended for full observability.
• Documentation: https://docs.langdb.ai
• Slack: https://join.slack.com/t/langdbcommunity/shared_invite/zt-2haf5kj6a-d7NX6TFJUPX45w~Ag4dzlg

Highlighted Details

• Supports multiple LLM providers including OpenAI, Gemini, Anthropic, DeepSeek, TogetherAI, XAI, and AWS Bedrock models.
• Offers OpenAI-compatible endpoints for chat completions, model listing, embeddings, and image generation.
• Features dynamic routing strategies for optimizing LLM selection based on cost, speed, or fallback mechanisms.
• Provides comprehensive usage analytics, cost tracking, rate limiting, and load balancing.

Maintenance & Community

The project is active with a public Slack channel for community support and discussion.

Licensing & Compatibility

Licensed under the Apache License 2.0, permitting commercial use and integration with closed-source applications.

Limitations & Caveats

The RUSTFLAGS with tracing_unstable and aws_sdk_unstable flags are required for cargo install, indicating potential instability or experimental features in those areas. The project also offers hosted and enterprise versions, suggesting the open-source version may lack certain advanced enterprise features or support levels.