This project provides LoRA fine-tuned weights for the ChatGLM-6B model, focusing on instruction following. It targets researchers and developers looking to enhance ChatGLM-6B's ability to understand and respond to instructions, particularly in Chinese. The primary benefit is an improved instruction-following capability for the base ChatGLM-6B model.

How It Works

The project leverages the LoRA (Low-Rank Adaptation) technique to fine-tune the ChatGLM-6B model on various instruction datasets. This approach injects trainable low-rank matrices into the existing model layers, significantly reducing the number of parameters that need to be updated during fine-tuning. This makes the fine-tuning process more memory-efficient and faster compared to full model fine-tuning, while still achieving competitive performance.

Quick Start & Requirements

• Install dependencies: pip install -r requirements.txt
• Requires 2x A100 80G GPUs for the DeepSpeed multi-GPU fine-tuning.
• Fine-tuned weights are available via Baidu Netdisk: https://pan.baidu.com/s/1c-zRSEUn4151YLoowPN4YA?pwd=hxbr (Extraction code: hxbr)

Highlighted Details

• Fine-tuned on 1 million Chinese instruction data generated by the BELLE project.
• Supports multi-GPU fine-tuning with DeepSpeed, offering an 8-9x speedup.
• Includes LoRA weights fine-tuned on Stanford Alpaca (52k English instructions) and BELLE (0.5M/1M Chinese instructions).
• Provides instructions for data preprocessing, including conversion to JSONL and tokenization.

Maintenance & Community

• The project references several other open-source projects and datasets, indicating community engagement.
• No explicit links to community channels (Discord/Slack) or a roadmap are provided in the README.

Licensing & Compatibility

• The README does not explicitly state a license for the project's code or the fine-tuned weights.
• The base model, ChatGLM-6B, is typically subject to its own license terms.

Limitations & Caveats

• The project is primarily focused on fine-tuning and does not include model evaluation methodologies.
• A GCC version upgrade might be necessary for certain environments.
• The availability of fine-tuned weights is through a third-party cloud storage service (Baidu Netdisk).