This repository provides Stable Diffusion models for high-resolution image synthesis, targeting researchers and developers interested in text-to-image generation. It offers various model versions, including SD 2.0 and 2.1, with capabilities for text-guided image generation, image modification, upscaling, and inpainting, built upon latent diffusion models.

How It Works

Stable Diffusion is a latent text-to-image diffusion model. It operates by conditioning a diffusion model on text embeddings derived from an OpenCLIP ViT-H/14 text encoder. The models are trained from scratch and utilize a v-prediction or noise-prediction approach within a U-Net architecture, operating in a lower-dimensional latent space for efficiency. This approach allows for high-resolution synthesis and various image manipulation tasks.

Quick Start & Requirements

• Installation: pip install -e .
• Dependencies: PyTorch 1.12.1, Transformers 4.19.2, Diffusers, invisible-watermark.
• Recommended: xformers for efficient attention on GPUs. Installation involves cloning the xformers repository, compiling it (up to 30 min), and installing.
• Intel CPU Optimization: Requires numactl, libjemalloc-dev, intel-openmp, intel_extension_for_pytorch.
• Model Weights: Must be downloaded separately from Hugging Face.
• Resources: Requires significant GPU VRAM for higher resolutions (e.g., 768x768).
• Documentation: Hugging Face for models, Project Page for SD 2.1.

Highlighted Details

• Offers SD 2.1 models at 768x768 and 512x512 resolutions.
• Includes depth-guided diffusion for structure-preserving image modification.
• Provides a text-guided x4 superresolution model.
• Features a text-guided inpainting model.
• Supports Intel® Extension for PyTorch* optimizations for CPU inference.

Maintenance & Community

The project acknowledges contributions from Hugging Face, LAION, and the DeepFloyd team. Codebase builds on OpenAI's ADM and lucidrains/denoising-diffusion-pytorch.

Licensing & Compatibility

• Code License: MIT License.
• Model Weights License: CreativeML Open RAIL++-M License. This license may have restrictions on commercial use and requires adherence to specific usage guidelines.

Limitations & Caveats

The models mirror biases present in their training data. The provided weights are research artifacts; use in production requires additional safety mechanisms. The README notes potential numerical instabilities with FP16 precision on the v2.1 model's vanilla attention module.