<2-3 sentences summarising what the project addresses and solves, the target audience, and the benefit.> IST-DASLab/llmq provides a high-performance, pure CUDA/C++ framework for training quantized Large Language Models (LLMs). It targets medium-sized, multi-GPU setups, enabling efficient LLM training and fine-tuning on accessible hardware with significant speed and memory benefits.

How It Works

This project implements LLM training in pure CUDA/C++, inspired by llm.c, focusing on efficient quantized training. It leverages low-bit precision for computations and optimizer states to reduce memory and accelerate training. The architecture supports multi-GPU parallelism and advanced techniques like activation recomputation and host memory offloading for maximizing model size and throughput.

Quick Start & Requirements

Builds require C++20, CUDA 12+, NCCL, cuDNN; CMake manages dependencies. Primary build: cmake -S . -B build && cmake --build build --parallel --target train. Data prep uses scripts/tokenize_data.py. Training requires specifying models, data, and optimization parameters. Dependencies include gcc-13, cmake, ninja-build, git, and specific CUDA/cuDNN/NCCL versions. Python 3.12+ for bindings/scripts.

Highlighted Details

• Performance: High TPS rates, with benchmarks showing over 300k TPS for smaller models on 4x RTX 4090 GPUs and high SOL utilization (up to 82%).
• Memory Optimization: Extensive control over activation recomputation (e.g., --recompute-block) and offloading of optimizer states/weights to host memory, enabling larger models.
• Quantization Flexibility: Supports low-bit data types for matrix multiplications (e4m3) and optimizer states (bf16), with options for persistent quantized weights.
• Python Bindings: Optional Python bindings available for integration with Python workflows, with pre-built wheels provided.

Maintenance & Community

Inspired by Andrej Karpathy's llm.c. No specific details on active maintainers, community channels, or a public roadmap are provided.

Licensing & Compatibility

The license type is not explicitly stated in the README, a critical omission for adoption decisions. Python bindings are built against specific CUDA versions (12.8, 13.0) and compute capabilities.

Limitations & Caveats

tokenize_data has limited dataset support. FP32 attention backward pass is non-deterministic. Python bindings offer coarse-grained operations and lack multi-process support. The absence of a stated license is a significant adoption blocker.