Summary

UniSD provides a unified framework for self-distillation (SD) in Large Language Models (LLMs), addressing fragmentation in existing methods. It enables efficient LLM adaptation without requiring stronger external teachers, offering improved performance and steerability for researchers and engineers.

How It Works

The framework integrates five complementary mechanisms: multi-teacher agreement (sequence/token-level), EMA teacher stabilization, token-level contrastive learning, feature matching, and divergence clipping. These components target supervision reliability, representation alignment, and training stability. The integrated UniSD* recipe combines these for optimal results, leveraging self-derived supervision.

Quick Start & Requirements

• Installation: Requires Python 3.12 and CUDA 12.8 (cu128 wheels). Installation involves environment setup, installing specific PyTorch/CUDA builds, compiling flash-attn, and then pip install -r requirements.txt --no-build-isolation.
• Prerequisites: Python 3.12, CUDA 12.x toolkit (e.g., 12.6), PyTorch 2.11.0 (cu128), flash-attn, vLLM, transformers. Optional: WANDB_API_KEY, HF_TOKEN.
• Setup Time: flash-attn compilation can take ~20 minutes initially.
• Links: Project Website: https://unifiedsd.github.io/, arXiv: https://arxiv.org/abs/2605.06597.

Highlighted Details

• Comprehensive study of self-distillation across supervision reliability, representation alignment, and training stability.
• Evaluated five mechanisms on 6 benchmarks, 6 models (Qwen2.5, Llama-3.1, Gemma-3, InternLM3), and 3 model families.
• UniSD* integrated recipe achieves state-of-the-art performance using only self-derived supervision.
• Supports diverse tasks including scientific reasoning, code generation, commonsense reasoning, and tool usage.

Maintenance & Community

Author affiliations include Georgia Institute of Technology, UCLA, and Carnegie Mellon University. No direct community links (e.g., Discord, Slack) are provided in the README.

Licensing & Compatibility

Released under the Apache License 2.0, permitting commercial use and integration with closed-source projects.

Limitations & Caveats

A potential compatibility warning exists between trl (v1.4.0) and vLLM (v0.20.2), with a recommendation to pin vllm < 0.19 if runtime errors arise, indicating possible integration fragility.