GaussianGPT introduces an autoregressive approach to 3D Gaussian scene generation, moving beyond diffusion or flow-matching methods. It enables generation, completion, and outpainting within a unified transformer-based framework, offering a novel paradigm for controllable and context-aware 3D content creation. The project targets researchers and practitioners in 3D computer vision and generative modeling.

How It Works

GaussianGPT employs a two-stage pipeline. First, a sparse 3D convolutional VQ-VAE compresses 3D Gaussian primitives into discrete latent tokens. These tokens are then serialized and modeled autoregressively by a causal transformer utilizing 3D rotary positional embeddings. This sequential generation process allows the model to naturally handle tasks like scene completion and outpainting by predicting subsequent tokens based on preceding ones, leveraging the compositional inductive biases of autoregressive models for explicit 3D representations.

Quick Start & Requirements

The environment requires CUDA 12.9 and PyTorch 2.8, with Python 3.10. Installation involves creating a conda environment and compiling several core dependencies from source, including gsplat, pytorch3d, flash-attention (v2.8.2), and MinkowskiEngine (requiring a specific CUDA 12 compatible fork and GCC <= 13). The primary installation command is pip install --no-build-isolation -r requirements.txt after setting up the environment and TORCH_CUDA_ARCH_LIST. Setup is described as "slow" due to compilation. Official resources include the Project Page, arXiv, and a Paper.

Highlighted Details

• Fully autoregressive generation of 3D Gaussian scenes.
• Unified model for scene generation, completion, and outpainting.
• Leverages a VQ-VAE for discretizing Gaussian primitives into tokens.
• Transformer architecture incorporates 3D rotary positional embeddings.
• Pre-trained checkpoints for scene-level (3D-FRONT, ASE) and object-level (PhotoShape) generation are available.

Maintenance & Community

The project is authored by Nicolas von Lützow, Barbara Rössle, Katharina Schmid, and Matthias Nießner, affiliated with TUM. Recent news includes acceptance to ECCV 2026 and the release of pre-trained checkpoints. No specific community links (Discord, Slack) or roadmaps are detailed in the README.

Licensing & Compatibility

This project is released under the MIT License, which is permissive and generally suitable for commercial use and integration into closed-source projects.

Limitations & Caveats

The installation process is complex, requiring specific versions of CUDA, PyTorch, and compilers (GCC <= 13 for MinkowskiEngine), and involves significant compilation time for CUDA extensions. The original 3D-FRONT dataset source is unavailable, necessitating the use of re-releases. Build failures are common and often related to CUDA extension compilation.