MTLA introduces a novel Multi-head Temporal Latent Attention mechanism designed to enhance efficiency in decoder-only architectures like LLMs. It achieves this by temporally compressing the key-value cache, significantly reducing memory footprint during inference, making it suitable for researchers and engineers working on large-scale speech and language processing tasks.

How It Works

MTLA builds upon DeepSeek MLA, incorporating temporal compression of the key-value cache. This core innovation allows for more efficient self-attention computations and a reduced memory overhead, particularly beneficial for autoregressive models. The library supports various attention mechanisms (MHA, MQA, GQA, MLA, MTLA) and positional encodings (RoPE, Decoupled RoPE).

Quick Start & Requirements

• Install the MTLA module: pip install mtla
• For full experiments, clone the repo and install fairseq: cd experiments/tools/fairseq && pip install --editable ./
• Prerequisites: PyTorch >= 1.10.0, Python >= 3.8.
• A Colab notebook demo is available for training and inference: https://colab.research.google.com/github/D-Keqi/mtla/blob/main/assets/MTLA.ipynb

Highlighted Details

• Supports multiple attention mechanisms: MHA, MQA, GQA, MLA, and MTLA.
• Includes setup recipes for speech translation (MuST-C), speech recognition (AMI), spoken language understanding (SLURP), and text summarisation (XSum).
• Offers Fairseq-style parallel beam search for evaluation and quality metrics like BLEU, WER, and ROUGE.
• Provides efficiency evaluation for inference time and GPU memory usage.

Maintenance & Community

The project is maintained by D-Keqi and Philip C. Woodland. Further community or roadmap information is not detailed in the README.

Licensing & Compatibility

The project does not explicitly state a license in the provided README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The README does not specify any limitations or known caveats. The project appears to be research-oriented with a recent arXiv publication date.