vLLM Router provides a high-performance, lightweight request forwarding system designed for large-scale vLLM deployments. It addresses the need for efficient load balancing and specialized routing, particularly supporting advanced techniques like prefill-decode disaggregation. This system is beneficial for engineers managing complex, high-throughput LLM inference infrastructure, offering enhanced control and performance optimization.

How It Works

The core architecture is a request routing framework built with async processing patterns. It implements multiple load balancing algorithms, including cache-aware, power of two, consistent hashing, random, and round robin, to distribute requests effectively. A key differentiator is its support for prefill-decode disaggregation, enabling specialized routing for separated inference phases, which can optimize throughput and latency. The system also integrates Kubernetes-native service discovery for worker management and health monitoring.

Quick Start & Requirements

• Primary Install: pip install vllm-router
• Prerequisites: Rust and Cargo (version checked via rustc --version, cargo --version), Python with pip.
• Build from Source: Requires cargo build --release for Rust components and python -m build for the Python package.
• Documentation: Usage examples for standard data parallelism and prefill-decode disaggregation are provided in the README.

Highlighted Details

• Supports advanced load balancing policies like cache-aware and power of two.
• Enables prefill-decode disaggregation for optimized inference pipelines.
• Features Kubernetes-native service discovery for worker management.
• Includes enterprise-grade features: circuit breakers, retry logic, and Prometheus metrics collection.
• Offers authentication via bearer token validation.

Maintenance & Community

This project is a fork of SGLang Model Gateway, with minimal changes at this stage. Further divergence is anticipated. No explicit community links (e.g., Discord, Slack) or details on notable contributors or sponsorships are provided in the README.

Licensing & Compatibility

The specific open-source license for this project is not explicitly stated in the provided README. This omission requires further investigation before adoption, especially concerning compatibility for commercial use or closed-source linking.

Limitations & Caveats

The README does not explicitly list known limitations, alpha status, or specific caveats. As a fork, its long-term maintenance trajectory and potential for breaking changes from the original SGLang Model Gateway are factors to consider. The absence of a stated license is a significant adoption blocker.