This repository serves as a curated collection of papers and blog posts focused on Large Language Model (LLM) based Long Context Modeling. It aims to be a comprehensive resource for researchers and practitioners interested in techniques that enable LLMs to process and understand extended sequences of text, covering areas like efficient attention mechanisms, memory augmentation, and length extrapolation.

How It Works

The repository organizes a vast and rapidly evolving field by categorizing relevant research papers and articles. It covers key sub-topics such as sparse and linear attention, recurrent transformers, state space models, retrieval-augmented generation (RAG), and various methods for compressing context or extending model context windows. The collection is regularly updated with recent publications, providing a dynamic overview of advancements in long context modeling.

Quick Start & Requirements

This is a curated list of research resources, not a software library. No installation or execution is required. The primary requirement is access to academic papers and online articles.

Highlighted Details

• Extensive categorization of papers across 16 major themes related to long context modeling.
• Regularly updated "News" section highlighting recently published papers.
• Includes links to a comprehensive survey paper and its associated repository.
• Provides bibtex citation for the survey paper.

Maintenance & Community

The repository is actively maintained by Xnhyacinth, with contributions welcomed via pull requests. It links to a related GitHub repository for further collaboration.

Licensing & Compatibility

The repository is licensed under the MIT License, allowing for broad use and distribution.

Limitations & Caveats

As a collection of links and summaries, the repository itself does not implement any models or code. The quality and accessibility of the linked papers depend on their original sources.