Summary

LLMIO is a Go-based load-balancing gateway designed to simplify the integration of multiple Large Language Model (LLM) providers. It offers a unified REST API, weighted scheduling, logging, and an administrative UI, enabling developers to manage and route requests to services like OpenAI, Anthropic, and Gemini through a single point of access. This project benefits users by abstracting provider-specific complexities and providing operational control.

How It Works

LLMIO functions as a central proxy, presenting a consistent API interface compatible with OpenAI Chat Completions, Gemini Native, and Anthropic Messages standards. It employs weighted scheduling strategies (random or priority-based) to distribute requests across configured LLM providers, allowing routing based on capabilities like tool calling or structured output. A modern React-based admin UI provides visibility into providers, models, logs, and metrics, while built-in rate limiting and connectivity checks enhance reliability. Configuration and request logs are persisted locally using a pure Go SQLite implementation.

Quick Start & Requirements

• Primary Install: Docker Compose (recommended) or Docker run. A local binary is also available via release downloads.
• Prerequisites: Docker, Go (>= 1.26.1 for development), pnpm (for frontend development). The TOKEN environment variable is required for public API access.
• Links: QQ group: 1083599685. Release downloads are available from the project's releases page.

Highlighted Details

• Unified API: Supports OpenAI, Gemini, and Anthropic message/completion formats, including streaming.
• Advanced Scheduling: Offers weighted random and priority-based scheduling, with routing logic adaptable to tool calling and structured output.
• Admin UI: A React, TypeScript, Tailwind, and Vite-based console for managing LLM infrastructure.
• Local Persistence: Utilizes an embedded SQLite database (db/llmio.db) for configuration and logs, requiring no external database setup.

Maintenance & Community

The project maintains a QQ group (1083599685) for community interaction. No specific details on core contributors, sponsorships, or a public roadmap are provided in the README.

Licensing & Compatibility

Released under the MIT License, which is permissive for commercial use and integration into closed-source applications.

Limitations & Caveats

The TOKEN environment variable is mandatory for public API access. Development requires a recent Go version (>= 1.26.1) and pnpm for the frontend. No explicit limitations regarding supported LLM providers beyond those listed are detailed.