Material Palette extracts PBR material properties (albedo, normals, roughness) from a single real-world image. It is designed for researchers and developers in computer vision and graphics seeking to generate physically-based rendering assets from existing imagery. The primary benefit is automating the creation of detailed material maps, reducing manual effort.

How It Works

The method operates in three stages: concept extraction from user-defined image masks, texture generation using these concepts, and decomposition into SVBRDF maps. This approach leverages diffusion models for high-quality texture synthesis and a dedicated decomposition network to derive material properties, offering a comprehensive pipeline from image to PBR assets.

Quick Start & Requirements

• Install via git clone and conda env create --verbose -f deps.yml.
• Requires Python 3.10.8, PyTorch 1.13, diffusers 0.19.3, peft 0.5, PyTorch Lightning 1.8.3.
• For decomposition, download pre-trained weights (model.tar.gz).
• Experiment with generation using pre-trained LoRA checkpoints (e.g., blue_tiles.zip).
• Full pipeline example requires an input image and a masks/ subdirectory with region masks.
• Project page: https://astra-vision.github.io/MaterialPalette/
• Paper: https://github.com/astra-vision/MaterialPalette/releases/download/preprint/material_palette.pdf

Highlighted Details

• Extracts albedo, normals, and roughness maps.
• Supports texture generation at various resolutions (1K, 2K, 4K, 8K).
• Offers concept learning via user-provided masks.
• Includes a pre-trained decomposition model for SVBRDF map generation.

Maintenance & Community

• Project is associated with CVPR 2024.
• Code includes components from PEFT, SVBRDF-Estimation, and DenseMTL.
• Visualizations utilize DeepBump and Blender.
• Questions can be posted via the issues tracker.

Licensing & Compatibility

• Released under the MIT License.
• Permissive for commercial use and closed-source linking.

Limitations & Caveats

The implementation does not include LPIPS filtering for generation ranking and outputs a single sample per region. Users may need to experiment with different prompts and parameters for optimal results.