This project provides an efficient GPU implementation for LLM inference using six-bit (FP6) quantization, targeting researchers and engineers seeking to reduce model size and inference costs while preserving accuracy. It offers significant speedups and memory reduction compared to FP16 and INT8 baselines.

How It Works

The core innovation is the TC-FPx system design, which enables Tensor Core support for various low-bit floating-point weights. It utilizes SIMT cores for runtime dequantization of x-bit weights to FP16 before feeding them to Tensor Cores for matrix multiplication. Ahead-of-time bit-level pre-packing optimizes memory access for irregular bit-widths, and SIMT-efficient runtime minimizes dequantization overhead.

Quick Start & Requirements

• Install via pip install .
• Requires PyTorch and CUDA.
• C++ API requires compilation via make.
• Tested on NVIDIA A100 GPUs; H100 and GH200 compatibility is expected.

Highlighted Details

• Achieves up to 8.9x speedup over bitsandbytes and 2.6x over FP16 baselines on linear layers.
• Near-lossless model compression with FP6, outperforming INT4 in quality.
• End-to-end inference on LLaMA-70b shows 1.69x-2.65x higher throughput, requiring fewer GPUs.
• Supports FP6_e3m2 and FP5_e2m2 weights with FP16 activations.

Maintenance & Community

• Project name recently changed back to QuantLLM.
• Paper accepted by USENIX ATC24.
• Integrated into DeepSpeed.
• Welcomes collaborations and community contributions.

Licensing & Compatibility

• No explicit license mentioned in the README.
• Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

Currently, FP6-LLM is primarily tested on A100 GPUs, and while other Tensor Core GPUs are expected to be compatible, further verification may be needed. The README mentions future support for FP4 and INT5, but these are not yet implemented.