LoRAMoE introduces a novel Mixture of Experts (MoE) approach designed to enhance language model alignment by preserving world knowledge. This method is particularly beneficial for researchers and practitioners working on large language model fine-tuning and alignment, aiming to mitigate knowledge degradation often seen in standard fine-tuning processes.

How It Works

LoRAMoE integrates a Mixture of Experts architecture directly into the LoRA (Low-Rank Adaptation) framework. By introducing localized balance constraints and a configurable number of experts, it allows specific parameters to be specialized while maintaining a global balance. This approach aims to retain the model's foundational knowledge by distributing expertise across multiple, potentially specialized, LoRA adapters, preventing catastrophic forgetting during alignment.

Quick Start & Requirements

• Installation: conda env create -f environment.yml or conda create -n loramoe python=3.10 -y followed by pip install -r requirements.txt. The peft package is not installed by default to avoid conflicts.
• Prerequisites: Python 3.10, Conda environment.
• Usage: Training is initiated via run_loramoe.sh. Evaluation is supported through OpenCompass.
• Documentation: Configuration details and hyper-parameter explanations are available within the README.

Highlighted Details

• Modifies transformers (specifically modeling_llama.py) and peft libraries to incorporate MoE into LoRA.
• Introduces hyper-parameters like blc_weight, blc_alpha, LoRA_rank, LoRA_alpha, LoRA_trainable, LoRA_dropout, and LoRA_num for expert configuration.
• Supports evaluation via OpenCompass by modifying its model configuration.

Maintenance & Community

The project is associated with authors from institutions including Shanghai Jiao Tong University and Alibaba Group. Citation details are provided for academic use.

Licensing & Compatibility

The repository does not explicitly state a license in the provided README.

Limitations & Caveats

The README notes that the peft package is intentionally excluded from the default installation to prevent conflicts with existing local installations, requiring manual management. The project's primary modifications are within the transformers and peft libraries, suggesting potential compatibility issues with future versions of these core libraries.