This extension provides ControlNet and T2I-Adapter capabilities for AUTOMATIC1111's Stable Diffusion WebUI, enabling users to guide image generation with various conditioning inputs like depth maps, Canny edges, and more. It targets users of Stable Diffusion WebUI seeking enhanced control over image composition, style, and structure, offering on-the-fly integration without model merging.

How It Works

The extension injects ControlNet models directly into the Stable Diffusion pipeline during inference, allowing for dynamic conditioning. It supports a wide array of preprocessors and ControlNet/T2I-Adapter models, including the latest ControlNet 1.1 features and T2I-Adapter models. A key advantage is its "Pixel-Perfect" mode, which automatically calculates optimal preprocessor resolutions to align with Stable Diffusion's internal processing, simplifying user workflow and improving output consistency.

Quick Start & Requirements

• Installation: Install via the "Install from URL" tab in the Stable Diffusion WebUI Extensions tab, using the URL https://github.com/Mikubill/sd-webui-controlnet.git. Restart the UI after installation.
• Models: Download ControlNet and T2I-Adapter models from the provided Hugging Face links and place them in the stable-diffusion-webui\extensions\sd-webui-controlnet\models directory.
• Prerequisites: Requires AUTOMATIC1111's Stable Diffusion WebUI. Minimum NVIDIA GPU: Ampere (4GB VRAM) with --xformers enabled and Low VRAM mode for resolutions up to 768x832. MacOS users may need --no-half.
• Documentation: ControlNet 1.1 Wiki

Highlighted Details

• Perfect support for all ControlNet 1.0/1.1 and T2I Adapter models and preprocessors.
• Enhanced compatibility with A1111's High-Res Fix, generating separate control images for base and high-resolution passes.
• Advanced Control Modes ("Balanced", "My prompt is more important", "ControlNet is more important") to fine-tune prompt vs. ControlNet influence.
• "Reference-Only" preprocessor allows image-based guidance without requiring specific control models.

Maintenance & Community

The project is actively maintained, with frequent updates adding new preprocessors and model support. Discussion threads for new features are linked in the README.

Licensing & Compatibility

The extension itself is typically distributed under a permissive license (e.g., MIT, Apache 2.0, depending on the specific fork/version, though the README doesn't explicitly state the license for this fork). However, it relies on Stable Diffusion models and ControlNet models, which have their own licenses. Compatibility for commercial use depends on the licenses of the underlying models used.

Limitations & Caveats

The "SD upscale" script is noted as buggy and not recommended; "Ultimate SD upscale" is preferred. Some features, like "CoAdapter", are not yet implemented. Using older ControlNet 1.0 models requires mapping old names (e.g., depth to depth_midas).