PiSSA is a parameter-efficient fine-tuning (PEFT) method for large language models that optimizes principal singular values and vectors, aiming for faster convergence and improved performance over methods like LoRA. It targets researchers and practitioners seeking efficient LLM adaptation.

How It Works

PiSSA adapts LLMs by focusing on the most significant components of weight matrices, identified via Singular Value Decomposition (SVD). Unlike LoRA, which updates a low-rank "noise" matrix added to the original weights, PiSSA directly optimizes the principal singular values and vectors, effectively freezing the less impactful parts of the model. This approach is claimed to lead to faster convergence and superior performance, particularly in quantized settings.

Quick Start & Requirements

• Install via pip install -r requirements.txt after cloning the repository and setting HF_ENDPOINT.
• Requires Python 3.10, CUDA 12.1, PyTorch 2.4.0, and flash-attn.
• Official documentation and datasets are available.

Highlighted Details

• Outperforms LoRA on five benchmarks, achieving 72.86% accuracy on GSM8K with Mistral-7B, compared to LoRA's 67.7%.
• Reduces 4-bit quantization error in LLaMA 2-7B by 18.97%, surpassing QLoRA and LoftQ on GSM8K.
• PiSSA initialization is fast, taking seconds via a quick SVD technique.
• Merged into the main peft library as an optional LoRA initialization method.
• Supports Conv2d and Embedding layers, with examples for SDXL.
• Integrates with DeepSpeed.

Maintenance & Community

• The project is associated with GraphPKU.
• Follow-up work and related research papers are listed, indicating active development in the field.

Licensing & Compatibility

• The repository does not explicitly state a license in the README.
• Compatibility with commercial or closed-source projects is not specified.

Limitations & Caveats

The README does not specify any limitations or known issues. The project appears to be actively developed, with recent updates adding support for new layer types and integrations.