This repository offers algorithms for latent KV cache compaction, specifically employing "Attention Matching" (AM) to reduce KV cache size while preserving attention behavior. It is designed for researchers and engineers working with large language models (LLMs) who need to optimize memory usage and inference speed, providing a method to significantly shrink KV caches without substantial performance degradation.

How It Works

Attention Matching (AM) compacts a KV cache by constructing a smaller set of keys and values in latent space that accurately reproduce the original attention output. This approach aims to maintain the model's performance by ensuring the compacted cache behaves similarly to the full cache during attention calculations, offering a novel way to manage memory-intensive KV caches.

Quick Start & Requirements

A quick demo can be run using python -m examples.qa_demo --model Qwen/Qwen3-4B --target-size 0.1. This command prefills a KV cache, compacts it to 10% using Attention Matching, and compares QA accuracy. The repository requires Python and specific model weights (e.g., Qwen3-4B). Detailed evaluation scripts are available in the evaluation/ directory. No direct links to official quick-start guides or demos are provided within the README.

Highlighted Details

• Implements "Attention Matching" (AM) for latent KV cache compaction, aiming for fast and effective size reduction while preserving attention behavior.
• Includes model-specific generation and caching utilities for Qwen3, Llama, and Gemma3.
• Provides tools for computing non-uniform head budgets to optimize compaction strategies.
• Features comprehensive evaluation scripts for QA and reasoning tasks to benchmark compaction methods.

Maintenance & Community

The project is under active development, with plans to evolve into a more polished, installable package. Contributions are welcomed via issues and PRs. A TODO list outlines current development plans. No specific community channels (e.g., Discord, Slack) or roadmap links are provided.

Licensing & Compatibility

The repository's README does not specify a software license. This absence of a declared license poses a significant adoption blocker, as it leaves the terms of use, modification, and distribution unclear, potentially restricting commercial or closed-source integration.

Limitations & Caveats

The project is explicitly marked as having "Development Status" and may be further developed into a more polished package. This suggests it may not be production-ready, and users should anticipate potential changes, bugs, or missing features as development continues, as indicated by the presence of a TODO list.