DeepSpec provides a comprehensive, full-stack codebase for training and evaluating draft models used in speculative decoding, a technique aimed at accelerating large language model inference. It caters to researchers and engineers seeking to optimize LLM performance by enabling the development and rigorous assessment of speculative decoding algorithms. The primary benefit is a streamlined workflow for creating and testing models that can predict and accept token generations, thereby reducing latency.

How It Works

The project follows a three-stage workflow: Data Preparation, Training, and Evaluation. Data preparation involves downloading prompts, regenerating target model answers, and constructing a potentially massive target cache. The Training stage utilizes this cache to train a draft model, with configurations specifying the algorithm (DSpark, DFlash, Eagle3) and target model. The Evaluation stage measures the draft model's acceptance rate on various benchmark tasks. This approach offers a unified framework for the entire speculative decoding development lifecycle.

Quick Start & Requirements

• Primary install: pip install -r requirements.txt.
• Prerequisites: An inference engine is required for data preparation. Default configurations assume a single node with 8 GPUs; adjustments are needed for fewer GPUs.
• Storage: The target cache can require substantial storage (e.g., ~38 TB for Qwen/Qwen3-4B).

Highlighted Details

• Supports three draft model algorithms: DSpark, DFlash, and Eagle3.
• Provides released checkpoints for target models like Qwen3 (4B, 8B, 14B) and Gemma (4-12B-it), paired with specific algorithms.
• Includes evaluation on diverse benchmarks such as GSM8K, HumanEval, MBPP, and MT-Bench.
• Emphasizes aligning setup with training settings for meaningful benchmark comparisons.

Maintenance & Community

Contributions of new algorithms are welcomed. Specific community channels (e.g., Discord, Slack) or roadmap links are not detailed in the provided README.

Licensing & Compatibility

DeepSpec is released under the MIT License. It incorporates code adapted from SpecForge (Apache-2.0) and DFlash (MIT), with full attribution provided in a NOTICE file. The MIT license generally permits commercial use and integration into closed-source projects, but users should consult the NOTICE file for specifics regarding adapted components.

Limitations & Caveats

The data preparation stage necessitates significant storage capacity for the target cache. Default scripts are optimized for multi-GPU environments (8 GPUs), requiring configuration adjustments for systems with fewer resources. For domain-specific applications or when the target model operates in "thinking mode," fine-tuning the draft model is recommended for optimal performance.