DiffusionLight presents a novel method for estimating lighting in single images by leveraging diffusion models to render a chrome ball. This technique targets researchers and practitioners in computer vision and graphics, offering a way to achieve convincing light estimates in diverse, real-world scenarios without relying on extensive HDR panorama datasets.

How It Works

The core innovation lies in using diffusion models, specifically fine-tuned Stable Diffusion XL with LoRA, to generate a chrome ball within an input image. A key discovery is the relationship between the diffusion noise map and chrome ball appearance, enabling consistent high-quality generation. The process involves inpainting a chrome ball, projecting it into an LDR environment map, and then composing multiple LDR maps via exposure bracketing to create an HDR light estimate. This approach bypasses the need for large, specialized datasets, improving generalization.

Quick Start & Requirements

• Install: conda env create -f environment.yml, conda activate diffusionlight, pip install -r requirements.txt
• Prerequisites: CUDA-toolkit and OpenEXR (if not using Conda). Input images should be resized to 1024x1024, padded if not square.
• Prediction:
  1. python inpaint.py --dataset <input_directory> --output_dir <output_directory>
  2. python ball2envmap.py --ball_dir <output_directory>/square --envmap_dir <output_directory>/envmap
  3. python exposure2hdr.py --input_dir <output_directory>/envmap --output_dir <output_directory>/hdr
• Links: Project Page, Paper, Colab, HuggingFace

Highlighted Details

• Addresses limitations of dataset-dependent methods by using general diffusion models.
• Fine-tunes Stable Diffusion XL with LoRA for exposure bracketing.
• Demonstrates superior generalization to in-the-wild scenarios.
• Provides evaluation code adapted from StyleLight and Editable Indoor LightEstimation.

Maintenance & Community

The project is associated with CVPR 2024. Further community or maintenance details are not explicitly provided in the README.

Licensing & Compatibility

The repository does not explicitly state a license. Users should verify licensing for commercial or closed-source use.

Limitations & Caveats

The README notes that diffusion models can sometimes insert incorrect or inconsistent objects. The process requires manual image resizing and padding.