MagicTime is a video generation pipeline designed for creating time-lapse videos, focusing on accurately depicting real-world processes with physical knowledge and strong variations. It targets researchers and developers in AI video generation, offering a novel approach to enhance model capabilities through a curated dataset and specialized training techniques.

How It Works

MagicTime leverages a DiT-based architecture, specifically integrating its methodology into the Open-Sora-Plan framework. The core innovation lies in its "metamorphic simulation" approach, trained on the ChronoMagic dataset, which comprises time-lapse video-text pairs. This dataset, augmented with detailed captions, enables the model to learn complex temporal dynamics and physical transformations, distinguishing it from general video generation models.

Quick Start & Requirements

• Installation: Clone the repository (git clone --depth=1), activate a conda environment (conda create -n magictime python=3.10.13, conda activate magictime), and install dependencies (pip install -r requirements.txt).
• Model Weights: Download weights from HuggingFace or WiseModel.
• Prerequisites: Python 3.10.13, Git LFS.
• Demos: Gradio Web UI (python app.py), CLI inference (python inference_magictime.py), Hugging Face Space, and Replicate demo are available.
• Resources: The project is built upon AnimateDiff and Open-Sora-Plan.

Highlighted Details

• Accepted by TPAMI (2025) and presented at arXiv (2404.05014).
• Offers multiple datasets: ChronoMagic (2.2K pairs), ChronoMagic-Bench (1.6K pairs), and ChronoMagic-Pro (460K pairs).
• Demonstrates superior performance in metamorphic video generation compared to models like MakeLongVideo, ModelScopeT2V, and VideoCrafter.
• Provides integration with DiT-based models like Open-Sora-Plan v1.0.0.

Maintenance & Community

The project is actively maintained by the PKU-YuanGroup, with contributions from community members providing extensions for ComfyUI and demos on Replicate and Jupyter Notebooks. Links to Twitter accounts of key contributors are provided.

Licensing & Compatibility

The project is primarily released under the Apache 2.0 license, allowing for commercial use and linking with closed-source projects.

Limitations & Caveats

The README notes that results can vary even with the same seed and prompt across different machines, suggesting potential non-determinism in the generation process. Training code is noted as "coming soon."