This project enables efficient inference of Mixtral-8x7B models on consumer hardware like Colab or desktops by offloading model experts between GPU and CPU memory. It targets researchers and developers working with large language models who need to run them on resource-constrained environments.

How It Works

The core approach combines mixed quantization using HQQ and a Mixture-of-Experts (MoE) offloading strategy. Different quantization schemes are applied to attention layers and experts to minimize memory footprint. Experts are offloaded individually and brought back to the GPU only when required, utilizing an LRU cache for active experts to reduce GPU-CPU communication during activation computation.

Quick Start & Requirements

• Run the demo notebook: ./notebooks/demo.ipynb
• No command-line script is currently available; the demo notebook serves as a reference for local setup.
• Requires Python and relevant ML libraries (specific versions not detailed in README).

Highlighted Details

• Efficient inference of Mixtral-8x7B on consumer hardware.
• Combines HQQ quantization with MoE offloading.
• Utilizes an LRU cache for active experts to optimize GPU-CPU communication.

Maintenance & Community

• Actively under development with plans to add support for more quantization methods and speculative expert prefetching.
• Contributions are welcomed.

Licensing & Compatibility

• License type not specified in the README.
• Compatibility for commercial or closed-source use is not detailed.

Limitations & Caveats

Some techniques described in the project's technical report are not yet implemented in the repository. The project is a work in progress, and a command-line interface for local execution is not yet available.