Summary

This repository provides a specialized, hardware-focused fork of the vLLM inference runtime, optimized for dual NVIDIA RTX 2080 Ti GPUs with NVLink. It enables users to run large language models, such as Qwen 27B, at high throughput (100+ tok/s decode) on older, cost-effective hardware. The project targets power users and researchers seeking to maximize inference performance on specific Turing-era NVIDIA hardware, offering a viable alternative to more expensive, newer GPUs.

How It Works

The project leverages a patched vLLM base (v0.21.0) with specific optimizations tailored for the SM75 architecture of dual RTX 2080 Ti cards. It integrates advanced techniques including Marlin for weight loading, FlashQLA/FlashInfer for efficient attention and quantized kernels, TurboQuant/INT8 KV for reduced memory bandwidth, and MTP/CUDAGraph for optimized execution. The core design prioritizes extreme single-concurrency performance, maximizing the combined 44GB VRAM and compute resources of the dual-GPU setup for demanding LLM inference tasks.

Quick Start & Requirements

• Build: Run ./build.sh to create a virtual environment, install dependencies, and build CUDA extensions.
• Run: Use ./launcher.sh for an interactive service manager (select checkpoints, profiles, modes, devices, ports) or scripted execution via environment variables.
• Prerequisites:
  • Dual RTX 2080 Ti 22GB GPUs with NVLink (PCIe P2P is a baseline requirement).
  • CUDA 12.8, PyTorch 2.11.0+cu128.
  • NVIDIA Driver 590.48.01 or compatible recent version.
  • Host OS: Ubuntu 22.04/24.04 LTS or Debian 12 with Linux kernel 6.x.
  • Host CPU: Modern with strong single-core performance; 16GB RAM recommended.
  • Sufficient SSD space for models and compile caches.
• Links: 2080Ti-LLM-Toolbox (companion toolbox).

Highlighted Details

• Achieves 100+ tok/s single-request decode for Qwen 27B FP8 on dual 2080 Ti with NVLink.
• Detailed comparison shows dual 2080 Ti (44GB VRAM, 1.62x CUDA cores, 3.24x Tensor cores) offers significant resource advantages over a single 3090 Ti (24GB) at roughly half the secondary-market price.
• Mature support for Qwen3.6 27B across various weight formats (FP8, INT4, NVFP4) and KV precisions (FP16, INT8, TurboQuant).
• Experimental support for Gemma4 31B, with noted limitations and ongoing validation.

Maintenance & Community

This project is a hardware-focused fork of upstream vLLM. It maintains the upstream project structure and integrates specific patches and validation notes. Notable upstream projects integrated include vLLM, FlashInfer, and FlashQLA. No specific community channels (Discord/Slack) or roadmap links are provided in the README.

Licensing & Compatibility

The project inherits the Apache-2.0 license from its upstream vLLM base. This license generally permits commercial use and modification, though specific implications of the fork's patches on compatibility with closed-source linking are not detailed.

Limitations & Caveats

This runtime is optimized for extreme single-concurrency performance and is not designed as a multi-tenant serving stack. While long prefill is supported, it may be serialized. Gemma4 31B support is experimental and has observed limitations in KV headroom and initialization. Compatibility with Turing GPUs other than the dual RTX 2080 Ti 22GB requires further validation regarding VRAM, P2P behavior, and specific runtime settings.