This repository introduces Lookahead Decoding, a novel parallel inference algorithm for Large Language Models (LLMs) that significantly accelerates generation speed without requiring a draft model or data store. It targets researchers and engineers seeking to optimize LLM inference latency, offering speedups of 1.5x to 2.3x.

How It Works

Lookahead Decoding adapts Jacobi Decoding, which treats LLM inference as solving nonlinear systems for simultaneous future token prediction. It enhances Jacobi's feasibility by caching and verifying n-grams generated from Jacobi iteration trajectories. The algorithm employs two parallel branches: a lookahead branch generates n-grams within a defined window (W, N parameters), and a verification branch uses string matching to identify and validate candidate n-grams via LLM forward passes, all optimized within a single attention mask.

Quick Start & Requirements

• Install via pip: pip install lade
• Install from source: git clone https://github.com/hao-ai-lab/LookaheadDecoding.git && cd LookaheadDecoding && pip install -r requirements.txt && pip install -e .
• Dependencies: Python, PyTorch. FlashAttention v2.3.3 is recommended for optimal performance.
• Demo: python minimal.py (with USE_LADE=1 LOAD_LADE=1)
• Docs: Paper, Blog

Highlighted Details

• Achieves 1.5x-2.3x latency reduction on various LLMs and datasets.
• Eliminates sequential dependency without draft models or data stores.
• Supports FlashAttention for further performance gains.
• Integrates into existing code with minimal changes (3 LoCs).

Maintenance & Community

• The project is associated with ICML 2024.
• Core implementation is in decoding.py, with model adaptations in models/.

Licensing & Compatibility

• The repository does not explicitly state a license in the README.

Limitations & Caveats

• Currently supports LLaMA models only.
• FlashAttention installation may require specific CUDA/PyTorch versions.