GaussianCube introduces a novel, structured, and explicit radiance representation for 3D generative modeling, addressing limitations of existing implicit or unstructured methods. It targets researchers and practitioners in 3D computer vision and graphics, enabling state-of-the-art results in unconditional, class-conditioned, and text-to-3D synthesis with significantly reduced parameter counts.

How It Works

GaussianCube first employs a densification-constrained Gaussian fitting algorithm for high-accuracy fitting with a fixed number of Gaussians. These Gaussians are then rearranged into a predefined voxel grid using Optimal Transport. This structured grid representation allows the use of standard 3D U-Net architectures in diffusion models without complex modifications, achieving high-quality representations with orders of magnitude fewer parameters than prior structured methods.

Quick Start & Requirements

• Install: Clone the repository, create and activate a conda environment using environment.yml.
• Prerequisites: Linux recommended, conda.
• Models: Download checkpoints and statistics from Hugging Face (links provided for Objaverse, OmniObject3D, ShapeNet).
• Inference: Run inference.py with specified model name and configuration.
• Docs: Paper, Project Page, Code

Highlighted Details

• Achieves state-of-the-art results in 3D generative modeling tasks.
• Offers 1-2 orders of magnitude parameter reduction compared to previous structured representations for similar quality.
• Supports text-conditioned generation, class-conditioned generation, unconditional generation, and digital avatar creation.
• Includes scripts for mesh conversion from generated results.

Maintenance & Community

• Codebase built upon improved-diffusion.
• Pretrained models and inference code released.
• Data construction and diffusion training code available.
• Project is associated with NeurIPS 2024.

Licensing & Compatibility

• The repository does not explicitly state a license in the README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

• Mesh conversion requires installing several additional dependencies, including custom builds like nvdiffrast and diff-gaussian-rasterization.
• Training data preparation requires specific formatting and pre-computation of statistics.