This repository is a curated reading list for individuals looking to deepen their understanding of Machine Learning Systems (MLSys). It provides a structured path through foundational papers, performance optimizations, quantization techniques, long context handling, sparsity, and distributed training strategies, aimed at engineers and researchers working with large-scale ML models.

How It Works

The list is organized thematically, guiding readers through key concepts in ML Systems. It starts with foundational attention mechanisms, progresses to performance optimizations like KV caching and continuous batching, covers quantization methods, and delves into advanced topics such as long context lengths, sparsity, and distributed training paradigms like tensor, pipeline, and data parallelism (ZeRO). The selection emphasizes papers that offer clear explanations and practical insights into system design and optimization.

Quick Start & Requirements

This is a reading list, not a software package. No installation or specific software requirements are needed beyond the ability to access and read research papers and potentially view associated code repositories. Links to papers and repositories are provided within the README.

Highlighted Details

• Covers foundational papers like "Attention Is All You Need" and "FlashAttention 2".
• Includes key performance optimizations such as KV caching, speculative decoding, and PagedAttention.
• Details various quantization techniques, including LLM.int8 and FP8 formats.
• Explores methods for handling long context lengths and sparsity.
• Discusses distributed training strategies like ZeRO and tensor/pipeline parallelism.

Maintenance & Community

This is a personal reading list, curated by an individual. There is no explicit mention of a community or ongoing maintenance beyond the initial curation.

Licensing & Compatibility

This repository contains links to external papers and code. The licensing of those individual resources would need to be checked separately. The list itself is not a software package and thus has no software license.

Limitations & Caveats

The list is a personal curation and may not be exhaustive. While it covers many critical areas, readers might need to consult additional resources for a complete understanding of the ML Systems landscape. The depth of coverage for each topic varies based on the selected papers.