This project delivers a from-scratch LLM inference engine in Rust and CUDA, meticulously optimized for NVIDIA Blackwell sm_120a GPUs (like the RTX 5090 Laptop). It prioritizes bit-exact inference and high performance on consumer hardware, offering a competitive alternative to established engines and enabling large model execution on limited VRAM.

How It Works

The engine employs custom-tuned CUDA kernels, focusing on bit-exactness verified through rigorous checks. Key innovations include NVFP4 (a 4-bit floating-point format) for efficient decoding, MTP (Multi-Token Prediction) speculative decoding, and advanced Mixture of Experts (MoE) handling with VRAM/host/disk spilling. Kernels are optimized against measured hardware limits, ensuring maximum throughput and accuracy on the target architecture.

Quick Start & Requirements

• Install: Build with cargo build --release.
• Run: Use provided executables (run-gen, run-spec, bw24-server).
• Prerequisites: NVIDIA Blackwell GPU (sm_120a), CUDA 12.8, Rust (2024 edition), cudarc 0.19.
• Models: Supports GGUF and NVIDIA safetensors checkpoints.
• Docs: ARCH.md, research/sm120-empirical-capabilities.md, research/benchmarks.md.

Highlighted Details

• NVFP4 decode path with int8 tensor-core GEMMs.
• MTP speculative decoding with K=1..8 self-consistency gate.
• MoE inference on 24 GB VRAM via disk-tier expert spilling.
• FlashAttention-style kernels with fused attention and quantized KV cache.
• Direct loading of safetensors checkpoints, including repacking for NVFP4.
• Sigmoid-router MoE support with disk-tier expert loading for models > RAM/VRAM.
• CUDA-graph decode for efficient per-token generation.

Maintenance & Community

Contributions are welcome via Issues and PRs, requiring adherence to correctness gates and performance measurement protocols. No community channels are specified.

Licensing & Compatibility

• License: MIT.
• Compatibility: Permissive for commercial use, but strictly hardware-bound to sm_120a GPUs.

Limitations & Caveats

The engine is specifically built and tuned for sm_120a GPUs; other architectures require a separate branch and may not be optimized. Model coverage is limited to tested formats and specific models, not a general runner. It operates on a single GPU and stream, lacking advanced parallelism features. As a research codebase, APIs and flags may change without notice.