This repository is a curated, non-comprehensive list of academic papers focused on Large Language Model (LLM) systems, maintained by ALCHEM Lab. It serves researchers and engineers interested in the latest advancements in LLM training, inference, and related system optimizations, providing a valuable resource for staying current in the rapidly evolving field.

How It Works

The repository categorizes papers into key areas: Algorithm-System Co-Design, LLM Inference (Serving) Systems, On-device LLM Inference, LLM Training Systems (Single-GPU and Distributed), General MLSys-Related Techniques, LLM Algorithm Papers, and Surveys. This structured approach allows users to quickly find relevant research on topics like Mixture-of-Experts (MoE), speculative decoding, parameter-efficient fine-tuning (PEFT), and various parallelism strategies.

Quick Start & Requirements

No installation or execution is required as this is a curated list of research papers. Links to papers are provided directly within the README.

Highlighted Details

• Extensive coverage of LLM inference systems, including papers on speculative decoding, efficient serving, and memory management (e.g., PagedAttention).
• Comprehensive collection of distributed training systems papers, featuring techniques like ZeRO, Megatron-LM, and various parallelism strategies (tensor, pipeline, expert).
• Inclusion of foundational LLM algorithm papers (e.g., "Attention is All You Need," LLaMA, PaLM) alongside system-focused research.
• Lists popular open-source LLM systems projects like vLLM, llama.cpp, and DeepSpeed.

Maintenance & Community

Maintained by ALCHEM Lab, the repository welcomes contributions via pull requests or issues for missing papers.

Licensing & Compatibility

The repository itself is a list of links to academic papers, each with its own licensing. The underlying research is generally accessible for academic and non-commercial use, with specific terms varying by publication venue.

Limitations & Caveats

The list is explicitly stated as non-comprehensive, meaning it may not cover every relevant LLM systems paper. The focus is on academic publications, and practical implementation details or code availability for each paper are not guaranteed within this list.