Summary

This repository introduces G-OPD, a generalized on-policy distillation framework for LLMs, addressing limitations in standard distillation. It enhances performance via reward scaling and flexible reference models, with the ExOPD variant outperforming standard OPD in same-size and strong-to-weak distillation. It targets researchers and engineers seeking improved LLM capabilities through knowledge transfer.

How It Works

G-OPD utilizes a novel on-policy distillation approach featuring reward scaling and a flexible reference model architecture for effective knowledge transfer. The framework supports single-teacher, multi-teacher (math/code domains), and on-policy context distillation (OPSD). OPSD leverages the student's in-context learning distribution as the teacher signal, using fixed teacher weights to prevent training collapse.

Quick Start & Requirements

Installation involves Conda setup with Python 3.10 (conda create -n verl python==3.10, conda activate verl), navigating to verl/, and running USE_MEGATRON=0 bash scripts/install_vllm_sglang_mcore.sh, followed by pip install math-verify. Prerequisites include verl v0.6.1, vllm, sglang, and mcore. Training data is available via a provided link; WANDB_API_KEY may be needed. Setup is complex, requiring significant computational resources (multi-GPU, large token lengths).

Highlighted Details

• ExOPD: Achieves state-of-the-art performance in on-policy distillation, outperforming standard OPD.
• Multi-Teacher Distillation: Supports two-teacher distillation, requiring explicit domain specification (e.g., math, code) and manual teacher model path configuration.
• On-Policy Context Distillation (OPSD): Enables self-distillation using the student's in-context learning outputs.
• Evaluation Suite: Comprehensive evaluation on Math Reasoning (AIME2024, AIME2025, DeepMath) and Code Generation (Absolute-Zero-Reasoner, EvalPlus, LiveCodeBench).

Maintenance & Community

No specific details regarding maintainers, community channels (e.g., Discord, Slack), or a public roadmap are provided in the README.

Licensing & Compatibility

The README does not specify the software license, potentially impacting commercial use or integration into closed-source projects.

Limitations & Caveats

OPSD avoids training collapse by using fixed student initial weights as the teacher, eschewing weight EMA. Multi-teacher distillation is limited to two teachers and requires manual setup. The absence of a specified license is a significant adoption caveat.