This repository provides Chinese fine-tuned versions of Meta's Llama 3 models, specifically addressing the need for high-quality Chinese language understanding and generation. It targets developers and researchers working with LLMs in Chinese, offering optimized conversational abilities, safety alignment, and extended context lengths.

How It Works

The project fine-tunes Meta's Llama 3 base models using a curated dataset of high-quality Chinese instruction data. This process enhances the models' performance on Chinese question answering, writing, and ensures alignment with safety and ethical guidelines. Notably, it offers versions with extended context windows (up to 28K tokens) for processing longer texts.

Quick Start & Requirements

• Install: pip install -r requirements.txt
• Prerequisites: Python 3.8+, transformers==4.37.0, torch==2.0.1, CUDA 11.7.
• Setup: Environment setup and model download are required.
• Links: Model download available on HuggingFace, ModelScope, and WiseModel.

Highlighted Details

• Offers 8B and 70B parameter models, including a 28K context length version.
• Fine-tuned for Chinese language, safety, and values.
• Supports standard Llama 3 chat templates and a custom llama3-unichat template for 8B models.
• Demonstrates capabilities with examples of factual recall, problem-solving, and safe response generation.

Maintenance & Community

• Developed by China Unicom AI Innovation Center.
• QQ group for technical discussion: 635964480.

Licensing & Compatibility

• Based on Meta Llama 3, which has its own license. Specific licensing for the fine-tuned weights is not explicitly stated but implies compatibility with Llama 3's terms.

Limitations & Caveats

• The 70B model uses the official Llama 3 template, while 8B models have a custom template.
• A 64K context version is planned for future release.