BricksLLM provides an enterprise-grade API gateway for managing Large Language Models (LLMs) in production. It offers fine-grained access control, cost, and rate limiting per API key, targeting developers and organizations needing to control and monitor LLM usage. The gateway supports major LLM providers like OpenAI, Azure OpenAI, Anthropic, and vLLM, along with custom deployments.

How It Works

BricksLLM acts as a central proxy, routing requests to configured LLM providers. It intercepts requests to enforce policies such as rate limits, cost controls, and access restrictions based on API keys and associated tags. The system leverages PostgreSQL for storing configuration and Redis for caching and session management, ensuring efficient policy enforcement and analytics.

Quick Start & Requirements

• Install/Run: Deploy via Docker using docker compose up within the BricksLLM-Docker repository.
• Prerequisites: Docker, PostgreSQL, Redis.
• Setup: Requires configuring provider API keys and creating Bricks API keys via provided curl commands or SDK integration.
• Docs: Environment variables, Admin Server Swagger Doc, Proxy Server Swagger Doc.

Highlighted Details

• Supports OpenAI, Azure OpenAI, Anthropic, vLLM, Deepinfra, and custom deployments.
• Features PII detection and masking, caching, request retries, and failover mechanisms.
• Offers cost analytics, request analytics, and Datadog integration.
• Provides model and endpoint access control.

Maintenance & Community

The project is actively maintained by bricks-cloud. Community channels are not explicitly mentioned in the README.

Licensing & Compatibility

The README does not specify a license. Compatibility for commercial use or closed-source linking is not detailed.

Limitations & Caveats

The project is presented as production-ready with a managed version available, but the open-source version's specific stability and support guarantees are not detailed. The README does not mention specific versioning beyond a latest tag for Docker images.