This extension integrates AnimateDiff, a powerful AI video generation model, into the AUTOMATIC1111 Stable Diffusion WebUI. It targets users of Stable Diffusion WebUI who want to generate animated GIFs and videos directly within their familiar interface, offering a streamlined workflow with advanced features like ControlNet integration and prompt travel.

How It Works

The extension achieves animation by inserting motion modules directly into the UNet model at runtime. This approach avoids the need to reload model weights, allowing for seamless integration and faster iteration. It supports various AnimateDiff models and adapters, including Motion LoRA, Hotshot-XL, and AnimateDiff V3, enabling flexible control over animation style and quality.

Quick Start & Requirements

• Install: Add the extension via the AUTOMATIC1111 WebUI's "Extensions" tab by providing the GitHub URL.
• Prerequisites: WebUI >= 1.8.0, ControlNet >= 1.1.441, PyTorch >= 2.0.0.
• Models: Requires specific Motion LoRA, Hotshot-XL, or AnimateDiff V3 Motion Adapter models from the maintainer's HuggingFace repository.
• Documentation: How to Use

Highlighted Details

• Integrates AnimateDiff with ControlNet for advanced animation control.
• Supports features like prompt travel, ControlNet V2V, and FreeInit.
• Offers runtime insertion of motion modules, avoiding model reloads.
• Provides support for AnimateLCM from MMLab@CUHK.

Maintenance & Community

The project is actively maintained, with recent updates (v2.0.0-a, v2.0.1-a) focusing on refactoring and new feature integration. The maintainer is actively developing the extension, aiming to incorporate new research until a superior open-source video model becomes available. They are open to feature requests and contributions.

Licensing & Compatibility

The extension is released under a non-commercial license. Commercial use requires direct contact with the maintainer via email.

Limitations & Caveats

Some advanced features, such as ControlNet per-frame mask and "reference control" features, are designated as "Forge only" due to maintenance complexities with the base WebUI. Users must use specific motion adapter models provided by the maintainer for optimal results.