Efficient long-context LLM serving is addressed by RedKnot, an acceleration integration built on SGLang. It targets engineers and researchers requiring high-throughput inference for models handling extensive contexts. RedKnot offers significant reductions in prefill FLOPs (50-70%) and TTFT speedups (1.35x-2.2x) for long contexts, while maintaining near-lossless accuracy.

How It Works

RedKnot optimizes LLM inference by employing head-aware KV reuse and token-selective Feed-Forward Networks (FFNs). Its core approach classifies attention heads into four categories (global, local, retrieval, dense) to tailor KV storage and reuse strategies. Offline KV caching combined with RoPE relocation allows selective recomputation of necessary tokens at serving time, ensuring numerical alignment. Elastic Sparsity implements token-selective FFNs, skipping computations for low-attention tokens. The SegPagedAttention runtime manages KV storage with per-head page tables and segmented KV stores, enabling different head classes to have distinct visible windows.

Quick Start & Requirements

Installation is performed via pip install -e "python[all]". Certain models, specifically Qwen3.5-MoE and DeepSeek-V4, necessitate transformers 5.x (a .venv_tf5 environment is provided); system transformers 4.57 is incompatible. Benchmarks utilize NVIDIA GPUs (e.g., L20Y). Official installation and quick-start guides are available via SGLang documentation.

Highlighted Details

• Achieves ~70% FLOPs reduction and ~1.9x TTFT speedup for Qwen3-32B at 32K context.
• Delivers ~54% compute savings and ~2.16x TTFT speedup for Qwen3.5-35B-A3B (MoE) at 64K context.
• Mistral-7B-Instruct-v0.3 sees ~50% FLOPs saved and ~1.35x TTFT speedup at 32K context.
• Accuracy remains near-lossless across benchmarks, though some degradation is noted for MoE sparsity at very long contexts.
• Built upon SGLang, inheriting features like RadixAttention, continuous batching, and quantization.

Maintenance & Community

A detailed phase plan is available at python/sglang/srt/layers/attention/redknot/ROADMAP.md. Beyond acknowledging SGLang and vLLM, specific community or contributor details are not elaborated in the README.

Licensing & Compatibility

The project is licensed under the Apache-2.0 license, which is permissive for commercial use and integration into closed-source projects.

Limitations & Caveats

RedKnot exhibits degraded performance (repeated tokens) for Llama-3.3-70B-Instruct on LongBench, with single-GPU INT4 configurations experiencing Out-Of-Memory errors and multi-GPU bf16 triggering cross-device issues; these are pending further investigation. The full Qwen 3.5 series and DeepSeek-V4 support are slated for future releases.