Summary

This project addresses the significant challenge of running massive 744B parameter Mixture-of-Experts (MoE) models, such as GLM-5.2, on consumer-grade hardware with limited RAM. It delivers a pure C, zero-dependency engine that streams model experts from disk, enabling frontier LLMs to run on machines with approximately 25 GB of RAM. The primary benefit is democratizing access to extremely large language models by drastically reducing hardware barriers.

How It Works

The core innovation lies in a single C file engine that loads the dense components of GLM-5.2 (around 9.9 GB) into RAM while streaming the ~370 GB of sparse experts from disk on demand. It employs aggressive int4 quantization, MLA attention with a compressed KV-cache, a DeepSeek-style router, and native MTP speculative decoding. This design minimizes the active memory footprint, making colossal models runnable through efficient disk I/O and optimized C kernels.

Quick Start & Requirements

• Install/Run: Build the engine and run self-tests with cd c && ./setup.sh. Inference is initiated via COLI_MODEL=/path/to/model ./coli chat. Model conversion from FP8 to the engine's int4 format requires Python with torch, safetensors, huggingface_hub, and numpy.
• Prerequisites: Linux (or WSL2), gcc with OpenMP, AVX2 instruction set support, a minimum of 16 GB RAM, and approximately 370 GB of storage for the int4 model on a local NVMe drive (ext4 filesystem recommended for performance). An optional CUDA Toolkit is needed for the experimental CUDA backend.
• Resource Footprint: The int4 model occupies ~370 GB on disk. The dense model components require ~9.9 GB of resident RAM, with peak RSS around 20 GB during chat sessions (auto-capped). Cold decode performance incurs ~11 GB of disk reads per token.
• Links: Pre-converted GLM-5.2 int4 models are available on Hugging Face (e.g., jlnsrk/GLM-5.2-colibri-int4).

Highlighted Details

• Enables running GLM-5.2 (744B MoE) on ~25 GB RAM.
• Pure C implementation with zero runtime dependencies.
• Experts are streamed from disk, minimizing RAM usage.
• Features int4 quantization, MLA attention, and MTP speculative decoding.
• Provides an OpenAI-compatible API via coli serve.
• Includes an experimental CUDA backend for resident tensors.

Maintenance & Community

This project is described as a "one-person project." Development is supported through community contributions via GitHub issues, particularly for providing benchmark numbers from diverse hardware, sponsoring development, or donating hardware. No dedicated community channels like Discord or Slack are explicitly mentioned.

Licensing & Compatibility

• License: The engine is licensed under Apache 2.0. The GLM-5.2 model weights are released under the MIT license.
• Compatibility: Both licenses are permissive, making the engine and weights suitable for commercial use and integration into closed-source projects.

Limitations & Caveats

Performance is heavily dependent on disk I/O, leading to slow cold start token generation (~0.05–0.1 tok/s on the developer's hardware). The experimental CUDA backend is not fully optimized for streaming experts and has specific requirements. The accuracy impact of int4 quantization requires further comprehensive benchmarking. The API is currently text-only, lacking support for advanced features like tools or multimodal inputs. Acceptable performance necessitates a local NVMe drive.