NVIDIA Cosmos Framework is an end-to-end system for training and serving advanced "world models," exemplified by the Cosmos 3 model family. It targets researchers and engineers working on Physical AI, offering a unified approach to process and generate diverse data modalities including language, images, video, audio, and action sequences. The framework aims to consolidate capabilities typically found in separate vision-language models, video generators, and simulators into a single, cohesive platform, accelerating development and deployment of complex AI systems.

How It Works

The framework is structured as a single Python package, cosmos_framework/, encompassing both training and inference functionalities. Training leverages distributed strategies like FSDP, TP, CP, and PP, supporting native DCP checkpoints with HuggingFace safetensors compatibility and adapters for JSONL, WebDataset, and LeRobot datasets. Inference is powered by backends such as Diffusers, Transformers, and vLLM, enabling both offline batch generation and online serving via Ray and Gradio. Cosmos 3 models utilize a Mixture-of-Transformers architecture, designed for flexible input-output configurations across modalities.

Quick Start & Requirements

Installation requires system dependencies like curl, ffmpeg, git-lfs, libx11-dev, tree, and wget. A curated PyTorch + CUDA environment is recommended, often starting from an NVIDIA NGC base image (e.g., nvcr.io/nvidia/pytorch:25.09-py3). The package is installed using uv, with specific groups for CUDA versions (e.g., uv sync --all-extras --group=cu130-train). Training recipes are tested on 8x H100 80 GB GPUs, but configurations are adjustable. A quick single-GPU inference command is provided. Comprehensive guides for setup, training, and inference are available at the project's repository: https://github.com/NVIDIA/cosmos-framework.

Highlighted Details

• End-to-end framework for training and serving omnimodal world models.
• Supports joint processing and generation of language, images, video, audio, and action sequences.
• Employs a unified Mixture-of-Transformers architecture for Physical AI applications.
• Features distributed training (FSDP, TP, CP, PP) and flexible inference backends (Diffusers, Transformers, vLLM).
• Includes dataset adapters for JSONL, WebDataset, and LeRobot, with safetensors checkpoint support.

Maintenance & Community

Information regarding specific contributors, community channels (e.g., Discord, Slack), sponsorships, or a public roadmap is not detailed in the provided README excerpt.

Licensing & Compatibility

The README excerpt does not specify the project's license type or any compatibility notes for commercial use or integration with closed-source projects.

Limitations & Caveats

The framework is designed for high-performance computing environments, with training recipes tested on multi-GPU setups (e.g., 8x H100 80 GB), indicating significant hardware requirements. Users must ensure their system meets unspecified "System Requirements" and select appropriate CUDA variants for installation.