sd-webui-text2video  by kabachuha

Stable Diffusion WebUI extension for text-to-video generation

created 2 years ago
1,321 stars

Top 31.0% on sourcepulse

GitHubView on GitHub
1 Expert Loves This Project
Project Summary

This extension integrates text-to-video diffusion models like ModelScope and VideoCrafter into the AUTOMATIC1111 Stable Diffusion WebUI. It enables users to generate videos from text prompts, animate existing images, and loop videos, leveraging existing WebUI dependencies for a streamlined experience.

How It Works

The extension supports multiple text-to-video models, including ModelScope and VideoCrafter, by utilizing their PyTorch implementations. It allows for fine-tuning with LoRAs trained via specific repositories and offers optimizations like Torch2 and xformers for improved VRAM efficiency, enabling longer video generation on consumer hardware.

Quick Start & Requirements

  • Install via the AUTOMATIC1111 WebUI extension manager.
  • Requires Stable Diffusion WebUI.
  • Model weights must be downloaded manually and placed in specific subdirectories (e.g., stable-diffusion-webui/models/ModelScope/t2v).
  • Recommended VRAM: 6GB for ModelScope (256x256), 12GB+ for longer/higher-res videos. VideoCrafter requires ~9.2GB.
  • Official Docs: https://github.com/kabachuha/sd-webui-text2video

Highlighted Details

  • Supports ModelScope and VideoCrafter models.
  • LoRA support for custom fine-tuned models.
  • In-painting and video looping capabilities.
  • Torch2/xformers optimizations for VRAM efficiency.
  • WebAPI available.

Maintenance & Community

  • Project was previously unmaintained but is now actively maintained.
  • Seeking community contributions, especially pull requests.
  • Discord contact: @kabachuha

Licensing & Compatibility

  • The extension itself appears to be MIT licensed.
  • Model weights are subject to their respective licenses (e.g., ModelScope, VideoCrafter). Compatibility for commercial use depends on the underlying model licenses.

Limitations & Caveats

VideoCrafter support is noted as Work-In-Progress and requires more maintenance. Prompt weighting is currently only implemented for ModelScope.

Health Check
Last commit

1 year ago

Responsiveness

Inactive

Pull Requests (30d)
0
Issues (30d)
0
Star History
9 stars in the last 90 days

Explore Similar Projects

Feedback? Help us improve.