This repository provides SliceGPT, a post-training sparsification technique for transformer models, including Large Language Models (LLMs). It enables users to reduce model size and improve inference speed by applying orthogonal transformations and slicing weight matrices, without altering the model's architecture. The primary audience includes researchers and practitioners seeking to optimize transformer deployments.

How It Works

SliceGPT applies orthogonal transformations to each transformer layer, followed by slicing off the least significant rows and columns of weight matrices based on eigenvalue decay. This process replaces dense weight matrices with smaller ones, reducing embedding dimensions and thus memory footprint and latency. The method is designed to maintain model performance while achieving significant compression.

Quick Start & Requirements

• Install: pip install -e .[experiment]
• Prerequisites: CUDA-enabled GPU, Python. Hugging Face authentication may be required for certain models.
• Documentation: Hugging Face discussion

Highlighted Details

• Reduces model size and memory footprint.
• Achieves inference speedups without code optimization.
• Supports recovery fine-tuning (RFT) for performance restoration.
• Integrates with LM Eval Harness for evaluation.

Maintenance & Community

• Developed by Microsoft.
• Contributions are welcomed, subject to a Contributor License Agreement (CLA).
• Follows the Microsoft Open Source Code of Conduct.

Licensing & Compatibility

• The repository does not explicitly state a license in the provided README.

Limitations & Caveats

• Adding support for new models requires implementing custom model and layer adapters.
• Hugging Face authentication might be necessary for specific models.