This repository provides the official implementation for Inversion-Based Style Transfer with Diffusion Models (InST), a CVPR 2023 paper. It enables users to capture and transfer the artistic style of a single painting to new images without requiring complex textual descriptions, addressing limitations of previous methods in controlling shape and conveying elements. The target audience includes researchers and artists interested in novel image generation and style transfer techniques.

How It Works

InST leverages pre-trained text-to-image diffusion models by treating the artistic style of a painting as a learnable, implicit textual description. The core idea is to invert the diffusion process for a given style image, effectively learning its unique characteristics. This learned style representation then guides the diffusion model's synthesis process, allowing for style transfer while preserving content and structure. An optional AdaIN module is available for enhanced control.

Quick Start & Requirements

• Install via conda env create -f environment.yaml and conda activate ldm.
• Requires a pre-trained Stable Diffusion Model saved at ./models/sd/sd-v1-4.ckpt.
• Training involves running python main.py --base configs/stable-diffusion/v1-finetune.yaml -t --actual_resume ./models/sd/sd-v1-4.ckpt -n <run_name> --gpus 0, --data_root /path/to/directory/with/images.
• Generation is performed using InST.ipynb.
• Official documentation and comparison data are available via links in the README.

Highlighted Details

• Captures and transfers complete artistic style from a single painting.
• Avoids complex textual descriptions for style guidance.
• Preserves content and shape while transferring style.
• Demonstrates quality and efficiency across various artistic styles.

Maintenance & Community

The project is associated with the CVPR 2023 paper "Inversion-Based Style Transfer With Diffusion Models". Contact information for questions and support is provided via email.

Licensing & Compatibility

The repository's license is not explicitly stated in the provided README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The README does not specify any explicit limitations or caveats regarding the implementation's status (e.g., alpha/beta), known bugs, or unsupported platforms. The setup requires downloading a specific pre-trained Stable Diffusion model.