MLE-LLaMA enhances the LLaMA model for improved Chinese language understanding and generation. It targets researchers and developers working with multilingual NLP, offering a fine-tuned LLaMA model with better Chinese fluency and instruction-following capabilities.

How It Works

The project leverages LLaMA's existing tokenizer, which naturally supports Chinese characters. Fine-tuning is achieved through two methods: a full fine-tuning script requiring significant GPU resources (80G A100) and a LoRA fine-tuning script using PEFT. The process involves a custom English-Chinese alignment dataset and instruction tuning using Chinese Alpaca and Guanaco datasets.

Quick Start & Requirements

• Install/Run: Requires downloading original LLaMA checkpoints from Hugging Face and placing them in a ckpt directory. Fine-tuning scripts (train.py or train_lora.py) are then executed.
• Prerequisites: Original LLaMA checkpoints, Python, PyTorch, PEFT. Full fine-tuning requires 80GB A100 GPU. LoRA fine-tuning parameters include batch size 128*8, 3 epochs, 256 cut length, and 2e-5 learning rate.
• Resources: Full fine-tuning is resource-intensive. LoRA fine-tuning achieves ~1.02s/it.
• Links: LLaMA, Stanford Alpaca, Hugging Face Language Modeling, Alpaca-LoRA, BELLE.

Highlighted Details

• Tokenizer naturally supports Chinese.
• Provides fine-tuning scripts for full fine-tuning and LoRA.
• Utilizes a fine-grained English-Chinese alignment dataset.
• Includes instruction tuning with Chinese Alpaca and Guanaco datasets.

Maintenance & Community

No specific information on contributors, sponsorships, or community channels is provided in the README.

Licensing & Compatibility

The README does not explicitly state a license for the MLE-LLaMA project itself. It relies on LLaMA, which has its own licensing terms. Compatibility for commercial use depends on the underlying LLaMA license.

Limitations & Caveats

The full fine-tuning script has very high hardware requirements (80G A100). The project notes that LLaMA tends to generate long sentences, which may persist.