This repository provides TRL-based code for reproducing the On-Policy Self-Distillation Fine-Tuning (SDFT) algorithm, a method designed to enable continual learning in foundation models. SDFT allows models to acquire new skills and knowledge from demonstrations without degrading existing capabilities, offering a practical solution when explicit reward functions are unavailable. It benefits researchers and engineers seeking to incrementally enhance foundation models while mitigating catastrophic forgetting.

How It Works

This repository implements On-Policy Self-Distillation Fine-Tuning (SDFT) using the TRL library. SDFT addresses continual learning challenges by enabling models to acquire new skills from demonstrations without degrading prior capabilities. It leverages in-context learning, using a demonstration-conditioned model as its own teacher to generate on-policy training signals. This approach circumvents the limitations of off-policy Supervised Fine-Tuning (SFT), consistently outperforming SFT in skill acquisition and knowledge tasks while substantially reducing catastrophic forgetting.

Quick Start & Requirements

• Installation: Clone the repository, set up a Python 3.12 virtual environment (conda or venv), and install dependencies via pip install -r requirements.txt.
• Execution: Run experiments using python main.py --model_name <path_to_model> --output_dir <output_path>.
• Prerequisites: Requires a single H200 GPU. Other hardware setups may necessitate refactoring and model size adjustments. The default model is Qwen/Qwen2.5-7B-Instruct.
• Links: Paper: https://arxiv.org/abs/2601.19897

Highlighted Details

• Implements the On-Policy Self-Distillation Fine-Tuning (SDFT) algorithm using the TRL library.
• Enables continual learning by acquiring new skills from demonstrations without performance regression.
• Consistently outperforms Supervised Fine-Tuning (SFT) in skill learning and knowledge acquisition tasks.
• Designed for reproduction on a single H200 GPU, with potential refactoring needed for other setups.

Maintenance & Community

The provided README does not contain information regarding notable contributors, sponsorships, community channels (e.g., Discord, Slack), or a public roadmap.

Licensing & Compatibility

The README does not specify the software license. Therefore, compatibility for commercial use or closed-source linking cannot be determined from the provided text.

Limitations & Caveats

The implementation is primarily geared towards reproducing research findings on specific hardware (single H200 GPU), and alternative setups may require significant refactoring or changes to model sizes. The repository's focus appears to be on experimental reproduction rather than a general-purpose library.