Wan-Dancer addresses the challenge of generating long-duration, high-quality, and rhythmically synchronized dance videos from music, a task difficult for current diffusion models due to temporal limitations. It offers researchers and developers a novel framework for creating minute-scale, coherent dance videos, enhancing creative AI applications.

How It Works

This project employs a hierarchical framework that decouples music-to-dance generation into global keyframe planning and local temporal refinement. It leverages full-track musical context to ensure long-range coherence, a significant improvement over methods limited to short sequences. Key innovations include dynamic frame rate adaptation using time-mapped RoPE embeddings for precise audio-video alignment, an optical-flow-based loss function for enhanced motion continuity, and motion-speed control to maintain high-fidelity details during rapid movements. This approach overcomes the typical 20-second duration barrier of diffusion models, enabling stable, coherent generation of extended dance sequences.

Quick Start & Requirements

• Primary install / run command: Installation involves setting up a Python 3.10.14 virtual environment, installing the package in editable mode (pip install -e .), and then installing specific dependencies like PyTorch 2.6.0+cu124, diffusers 0.34.0, and flash-attention 2.6.3.
• Non-default prerequisites and dependencies: Requires Ubuntu 22.04, Python 3.10.14, and CUDA 12.4. The hardware specification indicates a need for substantial GPU resources: 8 × NVIDIA A800 80GB GPUs.
• Estimated setup time or resource footprint: Setup involves multiple pip installs and specific CUDA versions, suggesting a moderate setup time. The hardware requirements represent a significant resource footprint.
• Links: Project Page: https://humanaigc.github.io/wan-dancer-project/, GitHub: https://github.com/Wan-Video/Wan-Dancer, MS Space: https://www.modelscope.cn/studio/model-detail/damo/cv_wan-dancer_video-generation, MS Model: https://www.modelscope.cn/models/damo/cv_wan-dancer_video-generation/overview, HF Model: https://huggingface.co/Wan-Video/Wan-Dancer, Paper: https://arxiv.org/abs/2607.09581.

Highlighted Details

• Generates stable, 720p/30fps videos exceeding one minute in duration.
• Demonstrates robust versatility across five distinct dance genres: Chinese Classic, K-Pop, Street, Tap, and Latin.
• Conditioned on both audio and textual prompts for flexible generation.

Maintenance & Community

The provided README does not detail specific contributors, sponsorships, or community channels like Discord or Slack. It notes integration with DiffSynth-Studio and Wan2.1.

Licensing & Compatibility

• License type: Apache 2.0.
• Compatibility notes: The Apache 2.0 license is generally permissive, allowing for commercial use and integration into closed-source projects.

Limitations & Caveats

The README focuses on the project's capabilities and does not explicitly list limitations or known bugs. However, the substantial hardware requirements (8x A800 80GB GPUs) present a significant barrier to entry for many users. The installation process also requires specific versions of PyTorch and other libraries, which may lead to dependency management challenges.