ContinualLM is an extensible PyTorch framework for continual learning in language models (LMs), addressing the challenge of acquiring new knowledge while retaining existing skills. It targets researchers and practitioners in NLP and continual learning, offering a unified pipeline for domain-adaptive pre-training and end-task evaluation.

How It Works

The framework focuses on domain-adaptive pre-training, treating each task as a domain-specific corpus. It implements a collection of state-of-the-art continual learning methods, including DAS, CPT, DGA, CTR, CLASSIC, B-CL, DEMIX, EWC, DER++, and HAT, alongside baselines like NCL, ONE, Adapter-ONE, Prompt-ONE, and KD. Evaluation is performed via end-task fine-tuning, providing a more robust assessment than perplexity.

Quick Start & Requirements

• Installation: conda create --name continuallm --file requirements.txt
• Dependencies: transformers==4.17.0, adapter-transformers==3.0.1.
• Example: A self-contained continual_pretrain.ipynb is available for soft-masking scenarios, runnable without GPUs.
• Dataset: A provided dataset with 6 domains and corresponding end-tasks is available for download.

Highlighted Details

• Offers checkpoints on Hugging Face for direct model usage and further pre-training.
• Includes a self-contained notebook example runnable without GPUs.
• Supports both domain-adaptive pre-training and end-task fine-tuning within a unified pipeline.
• Implements a wide range of established and novel continual learning methods for LMs.

Maintenance & Community

The project is associated with the UIC Liu Lab. Contact information for Zixuan Ke, Yijia Shao, and Haowei Lin is provided for inquiries.

Licensing & Compatibility

The repository's license is not explicitly stated in the README. Compatibility with commercial or closed-source projects is not specified.

Limitations & Caveats

The README mentions that using versions of transformers or adapter-transformers other than the specified 4.17.0 and 3.0.1 respectively may lead to unexpected bugs. The specific license for the codebase and its implications for commercial use are not detailed.