This repository provides a Python library for generating high-quality, smooth video transitions between text prompts using Stable Diffusion XL. It's designed for researchers and artists looking to create seamless visual narratives or dynamic animations with minimal computational cost and high user control.

How It Works

Latent blending manipulates intermediate latent representations within the diffusion process. It constructs a "diffusion tree" where branches are dynamically added based on perceptual similarity (LPIPS) to ensure smooth transitions. Key features include cross-feeding latents between branches to preserve structure and a novel approach to suppress motion artifacts, tricking the visual system into perceiving a single continuous image. SDXL Turbo support enables transitions faster than playback.

Quick Start & Requirements

• Install: pip install git+https://github.com/lunarring/latentblending
• Requirements: Python, PyTorch, CUDA (GPU required for reasonable performance).
• Optional Speedup: stable-fast compilation (do_compile=True).
• UI: python latentblending/gradio_ui.py
• Examples: single_trans.py, multi_trans.py

Highlighted Details

• Supports SDXL and SDXL Turbo for fast transitions.
• Dynamic branch injection using LPIPS for optimal smoothness.
• Customizable parameters: guidance scale, diffusion steps, cross-feeding power, range, and decay.
• Gradio UI for iterative keyframe generation and movie stitching.
• Time-based computation allows specifying compute budget instead of frame count.

Maintenance & Community

• Primary contact: Johannes Stelzer (@j_stelzer on Twitter).
• Future plans include macOS support, Huggingface Spaces integration, and ControlNet/IP-Adapter.

Licensing & Compatibility

• License: Not explicitly stated in the README. Compatibility for commercial use or closed-source linking is undetermined.

Limitations & Caveats

• macOS support is listed as "coming soon."
• Inpaint support has been dropped.
• The exact license is not specified, which may impact commercial adoption.