This repository provides code for LASER (Layer-Selective Rank Reduction), a method to improve Large Language Model (LLM) reasoning capabilities by replacing specific weight matrices with their low-rank approximations. It targets researchers and practitioners seeking to enhance LLM performance on tasks like question answering without extensive retraining.

How It Works

LASER intervenes in transformer layers by applying Singular Value Decomposition (SVD) to selected weight matrices, then reconstructing them using a specified fraction of the largest singular values. This process is controlled by three hyperparameters: the target layer (ℓ), the parameter type (τ, e.g., MLP or attention weights), and the rank retention fraction (ρ). This approach is advantageous as it can significantly boost performance with minimal computational overhead and no additional training.

Quick Start & Requirements

• Install dependencies: pip3 install -r requirements.txt
• Requires PyTorch and Hugging Face datasets and transformers.
• Example run command: python3 intervention_gptj_fever.py --lname fc_in --rate 9.9 --lnum 26
• Official website for results and discussions: https://pratyushasharma.github.io/laser/

Highlighted Details

• Achieves performance improvements on question-answering tasks without additional model training.
• Supports layer-selective intervention across MLP and attention weight matrices.
• Codebase includes scripts for reproducing paper results on various LLMs and benchmarks.
• Encourages community contributions for new LLM/dataset results to a public leaderboard.

Maintenance & Community

• The project is in early development with a planned major refactor in January 2024.
• Open to issues and pull requests.
• Discussions page available on the project website.

Licensing & Compatibility

• The repository does not explicitly state a license in the README.

Limitations & Caveats

• The code is described as an "early development release" and is undergoing refactoring.
• Adding support for new LLMs requires manual adaptation of the laser package and wrapper code.
• Some experiments may require separate hyperparameter selection on validation sets.