chatglm_finetuning  by ssbuild

Fine-tuning scripts for ChatGLM models

created 2 years ago
1,544 stars

Top 27.4% on sourcepulse

GitHubView on GitHub
Project Summary

This repository provides tools and scripts for fine-tuning the ChatGLM language model, specifically targeting the 6B parameter version and Alpaca-style fine-tuning. It is designed for researchers and developers looking to adapt large language models for specific tasks or datasets.

How It Works

The project leverages the deep_training library for distributed training and fine-tuning. It supports full parameter fine-tuning, LoRA, AdaLoRA, and IA3 adaptations, as well as PTV2 methods. The scripts facilitate data preparation, model training, and inference, enabling users to customize ChatGLM models with their own data.

Quick Start & Requirements

  • Install dependencies: pip install -U -r requirements.txt
  • Install deep_training: pip install -U git+https://github.com/ssbuild/deep_training.git
  • Core dependencies: transformers>=4.30, deepspeed, xformers, bitsandbytes>=0.39, accelerate>=0.20.
  • Model weights: glm-4-9b-chat, glm-4-9b-chat-1m.
  • Data format: JSON with id and conversations fields.
  • Training scripts: train_full.sh, train_lora.sh, train_ptv2.sh.
  • Inference scripts: infer.py, infer_finetuning.py, infer_lora_finetuning.py.
  • Official quick-start/docs: Not explicitly linked, but the README provides commands.

Highlighted Details

  • Supports multiple fine-tuning techniques: full parameter, LoRA, AdaLoRA, IA3, PTV2.
  • Includes scripts for data preparation, training, and inference.
  • Provides example data formats for fine-tuning.
  • Offers links to related fine-tuning projects for other LLMs.

Maintenance & Community

  • The project appears to be actively maintained, with recent updates mentioned.
  • Links to related projects suggest a community focus on LLM fine-tuning.
  • No specific community channels (Discord/Slack) or roadmap are provided in the README.

Licensing & Compatibility

  • The README does not explicitly state a license. The presence of Hugging Face model links suggests compatibility with Hugging Face's ecosystem.

Limitations & Caveats

The README does not detail specific limitations, but the reliance on specific versions of libraries like transformers and bitsandbytes may indicate potential compatibility issues with newer versions. The project focuses on ChatGLM models, limiting its direct applicability to other architectures without modification.

Health Check
Last commit

4 months ago

Responsiveness

1 day

Pull Requests (30d)
0
Issues (30d)
0
Star History
3 stars in the last 90 days

Explore Similar Projects

Starred by Chip Huyen Chip Huyen(Author of AI Engineering, Designing Machine Learning Systems), Ying Sheng Ying Sheng(Author of SGLang), and
9 more.

alpaca-lora by tloen

0.0%
19k
LoRA fine-tuning for LLaMA
created 2 years ago
updated 1 year ago
Feedback? Help us improve.