LLaMA-Adapter provides efficient fine-tuning methods for LLaMA models, enabling instruction-following and multimodal capabilities with significantly fewer parameters and reduced training time. It targets researchers and developers looking to adapt large language models for specific tasks without the computational cost of full fine-tuning. The primary benefit is achieving comparable performance to fully fine-tuned models with a fraction of the resources.

How It Works

LLaMA-Adapter introduces lightweight adapter modules into the LLaMA architecture. These adapters, comprising techniques like zero-init attention, prefix tuning, and learnable gates, are the only components trained. This parameter-efficient approach drastically reduces the number of trainable parameters (e.g., 1.2M for V1) and training time (e.g., 1 hour for V1), while stabilizing early training stages with a novel zero-init attention mechanism.

Quick Start & Requirements

• Install: pip install -r requirements.txt followed by pip install -e . within a Conda environment.
• Prerequisites: PyTorch with CUDA, Python 3.8. Access to LLaMA model weights is required (official or unofficial).
• Setup: Detailed setup instructions are provided for V1; other versions require referring to specific model directories.
• Links: Web Demo, LLaMA-Adapter V2 Paper, ImageBind-LLM

Highlighted Details

• Achieves instruction-following capabilities comparable to Stanford Alpaca and Alpaca-LoRA with only 1.2M parameters and 1 hour of training.
• Extensible to multimodal inputs, including image, audio, and video, with versions like LLaMA-Adapter V2 multimodal.
• Supports integration with LangChain for building LLM applications.
• Offers various released models (V1, V2 dialog, V2 multimodal, V2.1 multimodal) and related projects like ImageBind-LLM and SPHINX.

Maintenance & Community

• Active development with releases of V2, V2.1, and related projects like LLaMA2-Accessory, OneLLM, and SPHINX.
• Contributors include Peng Gao, Jiaming Han, Renrui Zhang, and Aojun Zhou.
• Hiring announcements for researchers at Shanghai AI Lab.
• Citation details provided for both V1 and V2 papers.

Licensing & Compatibility

• The README does not explicitly state the license for the LLaMA-Adapter code itself. However, it relies on LLaMA weights, which have specific usage terms. Compatibility for commercial use depends on the LLaMA license.

Limitations & Caveats

• Requires access to LLaMA model weights, which may have usage restrictions.
• Setup for versions beyond V1 requires consulting specific model directories.
• The README mentions a positional bias in GPT-4 evaluations, which was to be updated in the paper.