Summary

This project addresses the challenge of running extremely large Mixture-of-Experts (MoE) Large Language Models on consumer and workstation GPUs, which typically lack the VRAM required by official checkpoints. It provides a patched version of vLLM with custom SASS kernels and novel quantization/memory management techniques, enabling users to deploy models like GLM-5.2 (753B) and DeepSeek-V4-Flash (159B) on hardware like RTX 5090s and RTX PRO 6000s.

How It Works

The core innovation lies in a custom vLLM patch featuring hand-written SM120 SASS kernels. It employs a 2-bit quantization scheme for MoE experts, coupled with an FP4 "delta cache" and "confidence gate" to restore precision dynamically where needed, mitigating the sign-asymmetry issues of naive 2-bit methods. A tiered expert residency system allows models to spill from GPU VRAM to pinned host RAM and then to NVMe storage, with the GPU acting as a cache. This approach is enabled by custom SASS kernels for 2-bit GEMM and FP4 delta GEMM, optimized for Blackwell Tensor Cores.

Quick Start & Requirements

Installation is primarily via Docker, using Dockerfile.sm120-v024 for the core serving image or Dockerfile.recipes for pre-configured model/hardware setups. Key requirements include consumer/workstation Blackwell GPUs (SM120 architecture, e.g., RTX PRO 6000, RTX 5090) and potentially NVMe storage for tiered model residency. Recipes download checkpoints from HuggingFace on first run. Official quick-start recipes and detailed methodology are available in the bench/recipes/ directory and docs/ folder.

Highlighted Details

• Enables serving massive MoE models (up to 1T parameters) on consumer/workstation GPUs, overcoming VRAM limitations.
• Achieves near-FP4 checkpoint quality using 2-bit experts with FP4 precision recovery.
• Implements tiered expert residency (GPU VRAM -> Host RAM -> NVMe) for models exceeding available VRAM.
• Provides critical fixes for vLLM v0.24.0 on SM120 hardware, which is broken in the official release.
• Supports MTP speculative decoding and agent-ready tool/reasoning parsing.

Maintenance & Community

No specific details regarding maintainers, community channels (e.g., Discord, Slack), sponsorships, or a public roadmap were found in the provided README.

Licensing & Compatibility

The repository's license is not explicitly stated in the README. While it patches an open-source project (vLLM), specific terms for commercial use or closed-source linking are not detailed.

Limitations & Caveats

This project is strictly limited to consumer/workstation Blackwell GPUs (SM120 architecture). The custom SASS kernels and underlying toolchain are specific to this architecture and not portable. Significant host RAM and fast NVMe storage are recommended for running the largest models via tiered residency. The official vLLM v0.24.0 release is noted as non-functional on SM120 hardware.