This project introduces Mixture-of-Transformers (MoT), a sparse and scalable architecture designed for native multimodal foundation models. It addresses the computational inefficiency of dense models by incorporating modality-aware sparsity, enabling efficient generation across text, images, and speech. MoT is targeted at researchers and engineers building advanced multi-modal AI systems who seek to reduce computational costs without sacrificing performance.

How It Works

MoT enhances standard Transformer architectures by introducing modality-specific parameters within non-embedding layers. The core innovation lies in ModalityUntiedFeedForward and ModalityUntiedAttention modules. The FFN component utilizes separate feed-forward experts for each modality, routed based on modality_masks. Similarly, the attention mechanism employs modality-specific query, key, value projections, and normalization layers before performing a global self-attention computation. This design allows for specialized processing of different data types while enabling cross-modal interactions, leading to significant FLOPs reduction and improved efficiency.

Quick Start & Requirements

• Prerequisite: Requires an existing Transformer model implementation.
• Dependencies: PyTorch is implied by the provided code snippets.
• Hardware: Performance profiling was conducted on AWS p4de.24xlarge instances with NVIDIA A100 GPUs.
• Links: Paper: https://arxiv.org/abs/2411.04996

Highlighted Details

• In text + image generation (Chameleon setting), MoT (7B) achieves dense baseline quality using only 55.8% of FLOPs.
• For text + image + speech generation (Chameleon setting), MoT (443M) matches dense speech quality with just 37.2% of FLOPs.
• In text autoregressive + image diffusion generation (Transfusion setting), MoT (7B) matches dense model quality using one-third of FLOPs.
• System profiling shows MoT (Chameleon, 443M) achieves dense-level image quality in 47% of wall-clock time and dense-level text quality in 75.6% of wall-clock time.

Maintenance & Community

No specific details regarding maintenance, community channels (e.g., Discord, Slack), or active contributors are provided in the README.

Licensing & Compatibility

• License: BSD-3-Clause.
• Compatibility: The BSD-3-Clause license generally permits commercial use and integration into closed-source projects without significant restrictions.

Limitations & Caveats

The provided README focuses on the architecture's implementation and benefits, offering code snippets rather than a ready-to-run package. Integration requires a foundational understanding of PyTorch and existing Transformer architectures. Specific performance claims are tied to particular hardware configurations.