This repository serves as a comprehensive survey of resource-efficient techniques for Large Language Models (LLMs) and multimodal foundation models. It targets researchers and engineers working with large-scale AI models, providing a structured overview of algorithmic and system-level innovations aimed at reducing computational, memory, and storage demands. The primary benefit is a consolidated reference for understanding and implementing efficiency strategies in foundation models.

How It Works

The survey categorizes research into several key areas: resource-efficient architectures (e.g., efficient attention mechanisms, dynamic networks, Mixture-of-Experts), resource-efficient algorithms (pre-training, fine-tuning, inference, model compression), and resource-efficient systems (distributed training, federated learning, serving on cloud/edge). It focuses on papers from top-tier CS conferences and arXiv, published primarily after 2020, to capture the latest advancements.

Quick Start & Requirements

This repository is a curated list of papers and does not require installation or execution. It serves as a reference guide.

Highlighted Details

• Extensive coverage of efficient attention mechanisms, including Longformer, BigBird, Reformer, and Mamba.
• Detailed sections on model compression techniques like pruning, quantization (e.g., GPTQ, AWQ, BitNet), and distillation.
• Comprehensive overview of distributed training strategies (e.g., ZeRO, FSDP, Alpa) and efficient serving systems (e.g., Orca, FlexGen, PagedAttention).
• Includes papers on multimodal foundation models and their efficiency challenges.

Maintenance & Community

The repository is actively maintained by UbiquitousLearning. Contributions are welcomed via GitHub Issues for paper suggestions.

Licensing & Compatibility

The repository itself contains links to research papers and code repositories, each with their own licenses. The survey content is likely under a permissive license, but users should verify individual paper/code licenses.

Limitations & Caveats

The survey explicitly excludes hardware design innovations. It focuses on algorithmic and system aspects, and the definition of "resource" is limited to physical resources, excluding data or privacy considerations.