<2-3 sentences summarising what the project addresses and solves, the target audience, and the benefit.> This project explores instruction-tuning Stable Diffusion to enable precise image editing via natural language prompts and input images. It targets researchers and developers seeking to enhance diffusion models' ability to follow specific image transformation instructions, offering a methodology inspired by FLAN and InstructPix2Pix for improved zero-shot image manipulation capabilities.

How It Works

This project implements instruction-tuning for Stable Diffusion, inspired by FLAN and InstructPix2Pix. The core approach involves creating an instruction-prompted dataset and conducting training using an InstructPix2Pix-style methodology. It leverages libraries such as diffusers, accelerate, and transformers to fine-tune pre-trained Stable Diffusion models, enabling them to interpret user instructions alongside input images for specific edits and image transformation operations.

Quick Start & Requirements

• Primary install / run command: Recommended to use a Python virtual environment. Install dependencies via pip install -r requirements.txt.
• Non-default prerequisites and dependencies: Experiments were conducted with PyTorch 1.13.1 (CUDA 11.6) and a single A100 GPU. xformers is recommended for memory-efficient training (or automatically used with PyTorch 2.0+).
• Links: Blog post detailing the work: https://huggingface.co/blog/instruction-tuning-sd.

Highlighted Details

• Provides example training scripts for "Cartoonization" and "Low-level image processing," with options to train from scratch or fine-tune from existing InstructPix2Pix models.
• Offers pre-trained models on the Hugging Face Hub, including instruction-tuning-sd/cartoonizer and instruction-tuning-sd/low-level-img-proc.
• Includes interactive demos on Hugging Face Spaces for immediate experimentation without local setup.
• Training is performed at 256x256 resolution, with the observation that this does not significantly impact end quality at 512x512 inference resolution.

Maintenance & Community

The project acknowledges helpful discussions with Alara Dirik and Zhengzhong Tu. Further details and results are available in a Hugging Face blog post. Pre-trained models and datasets are hosted on the Hugging Face Hub.

Licensing & Compatibility

The README does not explicitly state a software license. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

Training is performed at a resolution of 256x256, though inference can be done at higher resolutions. The project appears to be research-oriented, focusing on demonstrating the instruction-tuning methodology rather than providing a production-ready, fully supported library. Specific details on potential bugs or unsupported platforms are not provided.