A unified toolkit for multimodal model inference, evaluation, and post-training, TorchUMM aims to simplify fair and reproducible comparisons across diverse AI architectures. It targets researchers and engineers working with multimodal models, offering a single interface to streamline complex workflows and accelerate development.

How It Works

TorchUMM employs a pluggable backbone architecture, integrating 14 multimodal model adapters under a unified inference interface. Its config-driven design, utilizing YAML files, allows seamless switching between models and benchmarks without code modifications. For scalability, it offers cloud-native execution via Modal, simplifying deployment on cloud GPUs.

Quick Start & Requirements

Installation involves cloning the repository recursively (git clone --recursive), followed by pip install -e .. Model-specific dependencies must be installed separately using pip install -r model/<model_name>/requirements.txt. Key prerequisites include Python and PyTorch, with specific models potentially requiring different versions. Flash Attention is often required or beneficial, and pre-compiled wheels are recommended. Cloud execution is supported via Modal. Links to model-specific guides and documentation are available within the repository.

Highlighted Details

• Supports 14 multimodal model adapters with a unified inference interface.
• Integrates over 10 benchmarks covering generation, understanding, and editing tasks.
• Provides post-training capabilities including SFT, IRG, recA, UniCot, and UniGame.
• Emu3.5 leverages native vLLM integration for enhanced performance, achieving approximately 74 tokens/s on 2x A100-80GB GPUs.
• Extensive benchmark results are provided for generation, understanding, and editing tasks, enabling direct comparison of supported models.

Maintenance & Community

The project appears to be under active development as of July 2026, with continuous additions of new models and benchmarks. No specific community channels (e.g., Discord, Slack) or notable contributors/sponsorships are mentioned in the README.

Licensing & Compatibility

The provided README does not explicitly state the software license. This omission requires further investigation for compatibility with commercial or closed-source projects.

Limitations & Caveats

The Emu3 model is incompatible with the UEval benchmark due to its use of separate components for understanding and generation. Emu3.5 exhibits option position bias on the MMBench benchmark due to its architecture and the CircularEval protocol. Evaluation results for the WISE benchmark are systematically lower than published numbers when using Qwen2.5-VL-72B-Instruct as the judge instead of GPT-4o. The project is under active development, and benchmark results may be updated.