This repository provides loralib, a Python package implementing Low-Rank Adaptation (LoRA) for large language models. It enables efficient fine-tuning by injecting trainable low-rank matrices into pre-trained models, significantly reducing the number of trainable parameters and storage requirements without introducing inference latency. The target audience includes researchers and engineers working with large NLP models who need to adapt them to specific tasks efficiently.

How It Works

LoRA reduces trainable parameters by decomposing weight updates into two smaller, low-rank matrices. This approach drastically cuts down the memory footprint for fine-tuning and allows for rapid task switching. The library integrates seamlessly with PyTorch models, offering drop-in replacements for nn.Linear, nn.Embedding, and nn.Conv2d layers.

Quick Start & Requirements

• Install via pip: pip install loralib or pip install git+https://github.com/microsoft/LoRA
• Requires PyTorch.
• Supports adapting nn.Linear, nn.Embedding, and nn.Conv2d.
• See examples for NLU and NLG tasks: examples/NLU/, examples/NLG/

Highlighted Details

• Achieves comparable or superior results to full fine-tuning on GLUE and NLG benchmarks with a fraction of trainable parameters.
• Demonstrates significant parameter reduction: RoBERTa base LoRA uses 0.8M trainable parameters vs. 125M for full fine-tuning.
• Supports lora.MergedLinear for handling fused layers (e.g., QKV projections) and optional bias training.
• Merging LoRA weights with base weights occurs automatically during model.eval(), eliminating inference latency.

Maintenance & Community

• The project is associated with Microsoft.
• Contact information for core contributors is provided.
• Follows the Microsoft Open Source Code of Conduct.

Licensing & Compatibility

• The repository does not explicitly state a license in the provided README text. However, it is a Microsoft project, which typically uses permissive licenses like MIT. Further verification of the LICENSE file is recommended for commercial use.

Limitations & Caveats

• Currently only supports PyTorch.
• The initial release of the repository has been archived on the snapshot-9-15-2021 branch.
• Adapting fused layers requires using lora.MergedLinear or manually splitting them.