2D relighting diffusion model for single-image object relighting
Top 87.8% on sourcepulse
Neural Gaffer is an end-to-end 2D relighting diffusion model that accurately relights any object in a single image under various lighting conditions. It targets researchers and practitioners in computer vision and graphics, enabling applications like text-based relighting, object insertion, and serving as a prior for 3D relighting tasks.
How It Works
The model employs a diffusion-based approach to relight objects. It takes a single input image and target lighting conditions to generate relit versions. A key aspect is its ability to act as a prior for 3D relighting, directly relighting radiance fields without requiring inverse rendering, which is a novel application of diffusion models in this domain.
Quick Start & Requirements
conda
to create an environment, activate it, and then pip install -r requirements.txt
and pip3 install -U xformers==0.0.28 --index-url https://download.pytorch.org/whl/cu118
.Highlighted Details
Maintenance & Community
The project is associated with NeurIPS 2024. The primary contributor is Haian Jin. The README indicates a TODO list for future releases.
Licensing & Compatibility
The repository does not explicitly state a license. The codebase is built on top of Zero123-HF.
Limitations & Caveats
The model was trained at a 256x256 resolution, which limits its ability to preserve fine details and can lead to relighting failures. The VAE used in the base diffusion model struggles with identity preservation for detailed objects at this resolution. Finetuning at higher resolutions or using a more powerful diffusion model is suggested to mitigate this.
1 month ago
Inactive