This repository serves as a curated collection of resources on Mixture-of-Experts (MoE) models, targeting researchers and engineers interested in sparse activation techniques for large-scale deep learning. It provides a comprehensive overview of foundational papers, recent advancements, open-source models, system implementations, and applications across various domains, aiming to accelerate understanding and development in the MoE field.

How It Works

The collection categorizes resources into key areas: Open Models, Papers (including "Must Read" selections), MoE Models, MoE Systems, MoE Applications, and Libraries. This structure allows users to navigate the landscape of MoE research and development, from theoretical underpinnings and seminal papers to practical implementations and system-level optimizations. The inclusion of links to papers, code repositories, and system implementations facilitates direct engagement with the technologies.

Quick Start & Requirements

This is a curated list of resources, not a runnable codebase. Users will need to refer to individual project links for installation and execution instructions. Prerequisites will vary significantly depending on the specific model, system, or library being explored.

Highlighted Details

• Comprehensive listing of influential MoE papers, including foundational works like "Switch Transformers" and "GLaM."
• Coverage of open-source MoE models such as OLMoE, DeepSeekMoE, LLaMA-MoE, Mixtral, and OpenMoE.
• Inclusion of systems designed for efficient MoE training and inference, like Tutel, Mesh-TensorFlow, FastMoE, and DeepSpeed-MoE.
• Categorization of MoE applications across diverse fields, including NLP, computer vision, speech recognition, and reinforcement learning.

Maintenance & Community

The repository is maintained by XueFuzhao. As a curated list, community contributions are encouraged via stars and forks, but specific community channels or active development discussions are not detailed.

Licensing & Compatibility

The repository itself is not a software project with a license. The licenses of the linked papers, models, and libraries will vary and must be checked individually.

Limitations & Caveats

This repository is a static collection of links and does not provide runnable code or direct support for any of the listed projects. Users must independently evaluate and integrate the referenced resources. The rapidly evolving nature of MoE research means the list may not always reflect the absolute latest advancements.