Fine-tuning scripts for ChatGLM models
Top 27.4% on sourcepulse
This repository provides tools and scripts for fine-tuning the ChatGLM language model, specifically targeting the 6B parameter version and Alpaca-style fine-tuning. It is designed for researchers and developers looking to adapt large language models for specific tasks or datasets.
How It Works
The project leverages the deep_training
library for distributed training and fine-tuning. It supports full parameter fine-tuning, LoRA, AdaLoRA, and IA3 adaptations, as well as PTV2 methods. The scripts facilitate data preparation, model training, and inference, enabling users to customize ChatGLM models with their own data.
Quick Start & Requirements
pip install -U -r requirements.txt
deep_training
: pip install -U git+https://github.com/ssbuild/deep_training.git
transformers>=4.30
, deepspeed
, xformers
, bitsandbytes>=0.39
, accelerate>=0.20
.glm-4-9b-chat
, glm-4-9b-chat-1m
.id
and conversations
fields.train_full.sh
, train_lora.sh
, train_ptv2.sh
.infer.py
, infer_finetuning.py
, infer_lora_finetuning.py
.Highlighted Details
Maintenance & Community
Licensing & Compatibility
Limitations & Caveats
The README does not detail specific limitations, but the reliance on specific versions of libraries like transformers
and bitsandbytes
may indicate potential compatibility issues with newer versions. The project focuses on ChatGLM models, limiting its direct applicability to other architectures without modification.
4 months ago
1 day