This repository is a curated list of efficient attention modules for Transformer models, aimed at researchers and engineers working with sequence modeling tasks. It provides a structured overview of various techniques designed to reduce the quadratic complexity of standard self-attention, enabling the processing of longer sequences and improving computational efficiency.

How It Works

The list categorizes efficient attention mechanisms based on their core ideas, such as using approximations (e.g., Reformer's LSH, Performer's random features), sparsity patterns (e.g., Longformer's global/blocked attention, Big Bird's random connections), or linearizations (e.g., Linformer's projection, Fast Transformers' kernel tricks). This approach allows users to quickly identify and compare different strategies for optimizing attention computation.

Quick Start & Requirements

This is a curated list, not a runnable library. To use any of the listed implementations, users must refer to the individual project repositories linked in the table. Requirements vary per implementation but generally include Python and deep learning frameworks like PyTorch or TensorFlow.

Highlighted Details

• Comprehensive table categorizing over 30 efficient attention modules.
• Includes links to papers, implementations, and computational complexity.
• Covers techniques like LSH, sparsity, linear approximations, and kernel methods.
• Lists relevant survey papers and benchmarks for efficient transformers.

Maintenance & Community

The list was last updated on March 10, 2021. There are no explicit mentions of active community channels or ongoing maintenance efforts within the README.

Licensing & Compatibility

The repository itself is a list and does not have a specific license. The licenses of the individual implementations linked within the list will vary and must be checked separately.

Limitations & Caveats

The list's last update was in early 2021, meaning it may not include more recent advancements in efficient attention mechanisms. The utility of this resource is dependent on the availability and maintenance status of the linked individual projects.