Cornucopia-LLaMA-Fin-Chinese provides instruction-tuned LLaMA models fine-tuned on Chinese financial knowledge. It targets users needing improved LLaMA performance in the financial domain, offering open-source, commercially usable models and a lightweight training framework.

How It Works

The project fine-tunes LLaMA-based models using instruction datasets derived from Chinese financial Q&A data, including publicly available and scraped sources. This approach aims to enhance LLaMA's capabilities in financial question answering by leveraging curated financial domain data and instruction-following techniques. Future work includes expanding datasets with GPT-4 and knowledge graphs for multi-task SFT and RLHF.

Quick Start & Requirements

• Install dependencies: pip install -r requirements.txt
• Install Git LFS for model downloads.
• Download LLaMA base models using ./base_models/load.sh.
• Inference scripts: ./scripts/infer.sh (single model), ./scripts/comparison_test.sh (multi-model).
• Finetuning script: ./scripts/finetune.sh (requires data in ./instruction_data/fin_data.json format).
• Python 3.9+ recommended.

Highlighted Details

• Offers LoRA weights for fine-tuned models based on decapoda-research/llama-7b-hf (V1.0) and Linly-AI/Chinese-LLaMA-7B (V1.1).
• Training requires significant GPU resources (e.g., A100-80GB) or high-end consumer GPUs (3090/4090 with 24GB VRAM) for batch sizes of 64-96.
• Demonstrates performance comparisons against original LLaMA, Wenxin Yiyan, and SparkDesk on financial Q&A tasks.
• Includes a comprehensive dataset construction methodology, with plans to use GPT-3.5/4.0 for further data enhancement.

Maintenance & Community

• Developed by Yangmu Yu and Wenhuan Hong from the Chinese Academy of Sciences, Chengdu Institute of Computer Applications.
• References include LLaMA, Stanford Alpaca, alpaca-lora, and Huatuo-Llama-Med-Chinese.
• Future development includes multi-task SFT, CUDA deployment, RLHF, and next-pretraining for Chinese financial domains.

Licensing & Compatibility

• Code License: Apache 2.0.
• Model weights are for academic research only and strictly prohibited for commercial use.

Limitations & Caveats

The project explicitly states that model resources are for academic research only and strictly prohibited for commercial use. The accuracy of model-generated content is not guaranteed due to computational factors, randomness, and quantization precision loss. Outputs should not be considered investment advice.