<2-3 sentences summarising what the project addresses and solves, the target audience, and the benefit.> This repository offers an experimental fork of llama.cpp focused on significantly enhancing LLM context window capacity within limited VRAM. It introduces Trellis-Coded Quantization (TCQ) for KV cache compression, enabling 2-3x more context while aiming to maintain or surpass FP16 quality. This is beneficial for users needing to process longer sequences or run larger models on constrained hardware.

How It Works

The core innovation is Trellis-Coded Quantization (TCQ), a novel approach to KV cache compression. Unlike standard methods that quantize values independently, TCQ constrains quantization indices to a trellis structure. This allows for a much larger effective codebook at low bitrates (2-3 bits per value). Combined with Fast Walsh-Hadamard Transform (FWHT) rotation and context-adaptive norm scaling, TCQ achieves substantial reductions in KL-divergence and perplexity, outperforming scalar quantization and even FP16 in some metrics.

Quick Start & Requirements

• Build: Requires CMake. For NVIDIA, use -DGGML_CUDA=ON. For AMD, use -DGGML_HIP=ON with specific ROCm targets.
• Prerequisites: NVIDIA GPU with CUDA toolkit for CUDA builds. ROCm 7.2+ and compatible AMD GPU for HIP builds.
• Recommended Configurations: turbo4 (4.25 bpv, ~3.8x compression), turbo3_tcq (3.25 bpv, ~5x compression, beats FP16 quality), turbo2_tcq (2.25 bpv, ~7x compression, max context).
• Example: ./build/bin/llama-server -m model.gguf -ngl 99 -fa -ctk turbo3_tcq -ctv turbo3_tcq

Highlighted Details

• Achieves 2-3x KV cache compression, enabling significantly larger context windows in the same VRAM.
• TCQ demonstrates lower perplexity than FP16 KV cache at 3.25 bits per value.
• Supports VRAM-constrained GPUs with --mmproj-gpu-swap for multimodal models, allowing CPU offloading of vision encoders.
• Compatible with any GGUF model where head_dim is a multiple of 128, with automatic zero-padding for others.

Maintenance & Community

This repository is described as a "highly experimental fork" of llama.cpp. Specific maintenance details, community links, or active contributor information for this fork are not provided in the README. Users should refer to the upstream llama.cpp project for broader community engagement.

Licensing & Compatibility

The specific license for this experimental fork is not explicitly stated in the README. It inherits from llama.cpp, which is typically distributed under the MIT license. However, users should exercise caution and verify licensing terms, especially for commercial or closed-source applications, given the experimental nature.

Limitations & Caveats

This is a "highly experimental fork" and should be used "at your own discretion." Users must ensure they have compatible NVIDIA (CUDA) or AMD (ROCm) hardware and the necessary build toolchains for compilation.