DreamLight addresses the challenge of harmonizing subjects with new backgrounds in images, offering both image-based and text-based relighting. It targets users needing to seamlessly composite elements while maintaining consistent lighting and color, benefiting from aesthetic uniformity and the flexibility of text-to-image generation for backgrounds.

How It Works

DreamLight reorganizes input data and leverages semantic priors from pre-trained diffusion models to generate natural relighting effects. It introduces a Position-Guided Light Adapter (PGLA) to condense light information into query embeddings, modulating the foreground with direction-biased masked attention. A Spectral Foreground Fixer (SFF) module adaptively reorganizes frequency components for enhanced foreground-background consistency.

Quick Start & Requirements

• Installation: pip install -r requirements.txt
• Prerequisites: PyTorch 2.4.1, CUDA 12.4. Requires specific versions of the diffusers library (0.33.1 for FLUX, 0.30.3 for SD1.5), which are included in the repository. Pre-trained model weights and CLIP image encoder weights must be downloaded separately and placed in ckpt/FLUX/ and ckpt/SD15/ folders.
• Usage: Navigate to the FLUX/ or SD15/ directory and run python test.py. Modify paths within test.py as needed. Single image inference requires specifying foreground, background, save paths, and a prompt text file.
• Links: Pretrained weights available via a provided link.

Highlighted Details

• Supports both image-based and text-based relighting.
• Utilizes a Position-Guided Light Adapter (PGLA) for directional light modulation.
• Includes a Spectral Foreground Fixer (SFF) for enhanced consistency.
• Offers versions based on FLUX (recommended for better performance) and SD1.5.

Maintenance & Community

• The project is presented as an arXiv preprint.
• Users are encouraged to report bugs.
• A Gradio demo is planned.

Licensing & Compatibility

• The license is not explicitly stated in the provided README.
• Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The README indicates that the diffusers library needs to be replaced with the versions provided within the repository, suggesting potential dependency management complexities. The project is presented as a research work, and its stability and long-term maintenance are not yet established.