DemoFusion provides a framework for generating high-resolution images using existing Latent Diffusion Models (LDMs), aiming to democratize access to advanced AI image generation. It is designed for researchers and users interested in pushing the boundaries of image resolution without requiring extensive computational resources for training.

How It Works

DemoFusion extends LDMs with three core mechanisms: Progressive Upscaling, Skip Residual, and Dilated Sampling. This approach allows for higher-resolution outputs by iteratively refining the image. The progressive nature also enables rapid prompt iteration by providing intermediate "previews" during generation.

Quick Start & Requirements

• Installation: Clone the repository and install dependencies via pip install -r requirements.txt.
• Prerequisites: Python 3.9+, CUDA (for GPU acceleration), PyTorch. A GPU with at least 18GB VRAM is recommended for high-resolution generation (e.g., 3072x3072), though a low-VRAM version is available requiring ~8GB VRAM.
• Demos: HuggingFace Spaces, Colab notebooks, and local Gradio demos are provided for Text2Image and Image2Image.
• Links:
  • HuggingFace Text2Image: https://huggingface.co/spaces/PRIS-CV/DemoFusion
  • HuggingFace Image2Image: https://huggingface.co/spaces/PRIS-CV/DemoFusion-Img2Img
  • Colab: https://colab.research.google.com/drive/1_f4055f7133034449847111771313313
  • Replicate: https://replicate.com/cjwbw/demofusion

Highlighted Details

• Achieves high-resolution image generation (e.g., 3072x3072) by extending open-source LDMs.
• Offers a low-VRAM version for users with limited GPU memory.
• Supports both Text2Image and Image2Image functionalities.
• Integrates with ControlNet for enhanced control over generation.

Maintenance & Community

The project was accepted to CVPR 2024. Community contributions have led to ComfyUI and Replicate integrations, as well as ControlNet and low-VRAM implementations.

Licensing & Compatibility

The repository does not explicitly state a license in the README. Users should verify licensing for commercial or closed-source use.

Limitations & Caveats

The Image2Image functionality is noted to have a strong bias correlated with SDXL's training data. Default hyper-parameters are recommended but may not be optimal for all use cases.