This project optimizes the inference speed of the Qwen3.5-122B-A10B large language model on NVIDIA DGX Spark hardware, achieving up to 52 tokens/second with a 256K context window. It targets researchers and engineers requiring high-throughput LLM deployment on specific NVIDIA Blackwell (SM121) architectures, offering a significant performance uplift over baseline configurations without quality degradation.

How It Works

The core approach combines several advanced optimization techniques within a modified vLLM framework. Key elements include leveraging the FlashInfer attention backend for improved memory hierarchy utilization, a hybrid INT4/FP8 quantization strategy for model weights, and MTP-2 speculative decoding to predict multiple tokens per step. Further acceleration is achieved via an INT8 quantized Language Model Head with Triton autotuning and a vLLM PR #38325 cherry-pick for optimized FP8 GEMM operations. An optional TurboQuant feature offers significant KV cache compression for increased concurrency at the cost of reduced throughput.

Quick Start & Requirements

Installation is automated via the ./install.sh script, which handles prerequisite checks and Docker image builds.

• Primary Install: bash ./install.sh
• Prerequisites: Docker, Python 3.12, torch, numpy, safetensors, huggingface_hub. Requires specific NVIDIA GB10 (Blackwell, SM121) GPUs and CUDA 13.0/13.2. Model checkpoints Intel/Qwen3.5-122B-A10B-int4-AutoRound and optionally Qwen/Qwen3.5-122B-A10B-FP8 are needed.
• Setup Time: Initial builds can take 30-60 minutes due to vLLM compilation for SM121. Subsequent runs are faster if Docker images are cached.
• Links: The repository itself serves as the primary documentation.

Highlighted Details

• Achieves 52 tokens/second (82% improvement) on Qwen3.5-122B-A10B on DGX Spark, with a LongCode peak of 54.9 tok/s.
• Supports a 256K context window with optimizations like TurboQuant KV cache compression for up to 5x capacity.
• Integrates multiple optimizations independently: FlashInfer, Hybrid INT4+FP8, MTP-2 Speculative Decoding, INT8 LM Head v2 with Triton autotune, and vLLM PR #38325 FP8 GEMM.

Maintenance & Community

The project acknowledges contributions from the vLLM community and specific projects like vllm-turboquant and spark_vllm_docker. While direct community links (Discord/Slack) are not provided, the acknowledgments suggest active community engagement and reliance on open-source components.

Licensing & Compatibility

The project is licensed under Apache 2.0, following the license of the original Qwen model. This license is permissive and generally compatible with commercial use and closed-source linking.

Limitations & Caveats

This work is specifically tested and optimized for a single NVIDIA DGX Spark node with GB10 GPUs. Multi-node or different hardware configurations are not covered and may require significant adaptation. The 256K context length is validated; performance and quality beyond this limit are unverified without custom calibration. The TurboQuant KV cache compression, while enabling higher concurrency, results in a ~22% reduction in generation speed.