Lit-LLaMA provides an independent, Apache 2.0 licensed implementation of the LLaMA language model, built on nanoGPT. It targets researchers and developers seeking to use, fine-tune, or pre-train LLaMA-compatible models without the GPL restrictions of the original Meta implementation, enabling broader integration and open-source AI development.

How It Works

This project offers a simplified, single-file implementation of LLaMA, prioritizing correctness and optimization for consumer hardware and scalable deployments. It leverages techniques like flash attention, INT8 and GPTQ 4-bit quantization for reduced memory footprint, and parameter-efficient fine-tuning methods such as LoRA and LLaMA-Adapter.

Quick Start & Requirements

• Install: pip install -e ".[all]"
• Requirements: Python, PyTorch. GPU with sufficient VRAM (e.g., ~26GB for 7B model without quantization, ~5GB with GPTQ int4). CUDA support is implied for GPU acceleration.
• Resources: Guide for generating samples: https://github.com/Lightning-AI/lit-llama/blob/main/README.md#generate-text-from-the-model
• Fine-tuning: Requires ~24GB GPU memory.

Highlighted Details

• Supports flash attention, INT8, and GPTQ 4-bit quantization.
• Enables LoRA and LLaMA-Adapter fine-tuning.
• Includes scripts for pre-training on datasets like RedPajama.
• Designed for numerical equivalence to the original LLaMA model.

Maintenance & Community

• Warning: This repository is no longer actively maintained. The successor project is LitGPT.
• Community: Discord server available for engagement. https://discord.gg/VptPCZkGNa

Licensing & Compatibility

• License: Apache 2.0.
• Compatibility: Permissive license allows integration with closed-source projects.

Limitations & Caveats

This repository is explicitly marked as "Not Actively Maintained." Users are directed to the LitGPT project for updated features and support.