Paint3D is a generative framework for creating high-resolution, lighting-less 2K UV texture maps for 3D meshes, conditioned on text or image inputs. It targets 3D artists and developers seeking to generate reusable, re-lightable textures for game development, animation, and virtual environments. The primary benefit is the ability to produce diverse, semantically consistent textures without baked-in lighting artifacts.

How It Works

Paint3D employs a coarse-to-fine diffusion model approach. It begins by generating view-conditional images using a pre-trained depth-aware diffusion model and fuses them into an initial coarse texture map. This stage addresses shape representation limitations of 2D models. Subsequently, specialized UV Inpainting and UVHD diffusion models refine the texture, filling incomplete areas and removing illumination artifacts in a shape-aware manner, resulting in high-quality, lighting-less textures.

Quick Start & Requirements

• Install: conda env create -f environment.yaml and pip install kaolin==0.13.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/{TORCH_VER}_{CUDA_VER}.html.
• Prerequisites: Tested on Centos 7 with PyTorch 1.12.1 and CUDA 11.6. Requires ControlNet models (downloadable from Hugging Face).
• Resources: Specific hardware requirements are not detailed, but diffusion models typically require significant GPU VRAM.
• Links: Project Page, Arxiv, Demo, FAQ.

Highlighted Details

• Generates 2K UV texture maps.
• Supports text and image conditioning.
• Produces lighting-less textures for re-lighting flexibility.
• Coarse-to-fine generative framework.
• Utilizes depth-aware diffusion and UV-specific refinement models.

Maintenance & Community

The project released its code in April 2024 and the paper in December 2023. Related projects include MVPaint and MeshXL. A ComfyUI node is available.

Licensing & Compatibility

Distributed under the Apache 2.0 LICENSE. Note that dependencies like PyTorch3D and PyTorch Lightning have their own licenses, which must also be followed.

Limitations & Caveats

The setup is tested on a specific older environment (Centos 7, PyTorch 1.12.1, CUDA 11.6), suggesting potential compatibility issues with newer setups. The README does not detail specific hardware requirements for running the models.