Generative 3D scene reconstruction is addressed by Lyra, which enables the creation of explicit 3D and 4D scenes from single images or videos. Targeting researchers and engineers in fields like gaming, robotics, and autonomous driving, it offers a novel approach that bypasses the need for extensive real-world multi-view data, facilitating faster development and deployment of virtual environments.

How It Works

The project employs a self-distillation framework to distill implicit 3D knowledge from powerful video diffusion models into an explicit 3D Gaussian Splatting (3DGS) representation. Lyra augments standard RGB decoders with a 3DGS decoder, which is trained using synthetic data generated by camera-controlled video diffusion models. This method allows for purely synthetic training, eliminating reliance on captured real-world data and leveraging the generative imagination of diffusion models for 3D scene synthesis.

Quick Start & Requirements

Installation involves setting up a conda environment and downloading pre-trained weights as per INSTALL.md. Key requirements include NVIDIA GPUs; testing was performed on H100 and A100. For memory-constrained GPUs, extensive offloading flags are available, with maximum observed inference memory around 43GB. Dynamic scene generation necessitates the ViPE tool, which requires a separate conda environment due to compatibility issues. Demo scripts are provided for single-image to 3D Gaussian reconstruction and single-video to dynamic 3D Gaussian reconstruction, involving latent generation followed by 3DGS decoding. Links to demo samples and configuration files are available within the repository.

Highlighted Details

• Achieves state-of-the-art performance in both static and dynamic 3D scene generation.
• Enables real-time rendering of synthesized 3D scenes from text prompts or single images.
• Extends generative capabilities to dynamic 4D scenes from monocular video input.
• Utilizes 3D Gaussian Splatting for efficient and high-fidelity scene representation.

Maintenance & Community

The project builds upon and draws inspiration from several open-source repositories, including NVIDIA Cosmos and GEN3C. Specific community channels like Discord or Slack are not detailed in the README.

Licensing & Compatibility

Lyra's source code is released under the permissive Apache 2 License. However, the Lyra models are distributed under the NVIDIA Open Model License, which may impose restrictions on commercial use or integration into closed-source projects. Custom licensing options are available through NVIDIA Research Licensing.

Limitations & Caveats

The system is primarily tested and optimized for high-end NVIDIA GPUs (H100, A100), with significant memory requirements even with offloading. The dependency on ViPE for dynamic scene generation requires careful environment management due to compatibility conflicts. The specific licensing for models may require review for commercial adoption.