This repository provides LLM-Adapters, a framework for parameter-efficient fine-tuning (PEFT) of large language models, extending HuggingFace's PEFT library. It targets researchers and practitioners seeking to adapt LLMs like LLaMA, OPT, BLOOM, and GPT-J to specific tasks using various adapter types, including LoRA, Bottleneck, Parallel, and Prefix Tuning, with demonstrated performance improvements on commonsense reasoning and arithmetic tasks.

How It Works

LLM-Adapters integrates multiple PEFT techniques by injecting adapter modules into the transformer architecture. It supports various adapter configurations, including LoRA, AdapterH (bottleneck), AdapterP, and Parallel adapters, allowing for flexible customization. The framework enables efficient fine-tuning by updating only these adapter parameters, significantly reducing computational cost and memory requirements compared to full fine-tuning.

Quick Start & Requirements

• Install dependencies: pip install -r requirements.txt
• Set BASE_MODEL environment variable (e.g., export BASE_MODEL=yahma/llama-7b-hf).
• Requires Python and PyTorch. GPU with CUDA is recommended for training and inference.
• Training example: torchrun --nproc_per_node=2 --master_port=3192 finetune.py --base_model 'yahma/llama-7b-hf' --data_path 'math_10k.json' --output_dir './trained_models/llama-lora' --adapter_name lora
• Inference example: torchrun generate.py --base_model 'yahma/llama-7b-hf' --lora_weights './trained_models/llama-lora'
• Evaluation example: python evaluate.py --model LLaMA-7B --adapter LoRA --dataset SVAMP --base_model 'yahma/llama-7b-hf' --lora_weights './trained_models/llama-lora'
• Paper: https://arxiv.org/abs/2304.01933

Highlighted Details

• Supports LLaMA, BLOOM, GPT-J, OPT, GPT-2, GPT-Neo, GPT-NeoX-20B, and ChatGLM models.
• Offers a comprehensive adapter support matrix, with some adapters in development for GPT-2 and GPT-NeoX-20B.
• Achieves competitive performance, with LLaMA-13B-Parallel outperforming ChatGPT on 8 commonsense benchmarks and LLaMA-13B-LoRA reaching 80.5% on average across math reasoning tasks.
• Includes custom datasets like math_10k.json and commonsense170k.

Maintenance & Community

• Project accepted to EMNLP 2023.
• Benefits from contributions and inspiration from PEFT, Adapter-Transformer, and Alpaca-lora.

Licensing & Compatibility

• The repository does not explicitly state a license. It is built upon HuggingFace's PEFT library, which is typically under the Apache 2.0 license. Compatibility for commercial use should be verified.

Limitations & Caveats

• Some adapter types are still under development for specific models (e.g., GPT-2, GPT-NeoX-20B).
• The README notes an apology for data collection issues with the math_10k.json dataset, affecting specific benchmarks.