This project provides a framework for generating interactive, browser-compatible 3D environments from single videos. It targets researchers and developers in computer vision and graphics, enabling the creation of realistic 3D scenes with features like mesh extraction, collision model generation, and texture baking.

How It Works

The system leverages Neural Radiance Fields (NeRF) for scene representation, extracting meshes and generating priors like depth and normals using external models (e.g., Omnidata). It supports semantic and instance segmentation for detailed mesh separation and texture completion, culminating in the generation of game-ready assets.

Quick Start & Requirements

• Install: Create a conda environment (conda create -n video2game python=3.7), activate it, and install dependencies including PyTorch (tested with CUDA 11.6, Torch 1.12.0), torch-scatter, tiny-cuda-nn, nvdiffrast, and pymesh.
• Prerequisites: Python 3.7, CUDA 11.6, PyTorch 1.12.0, git clone --recursive for tiny-cuda-nn, and potentially mmsegmentation for KITTI-360 data.
• Dataset Format: COLMAP format with optional normals/, depth/, and instance/ directories.
• Docs: Project Page

Highlighted Details

• Real-time, interactive, and browser-compatible environment generation.
• Supports mesh extraction with physical entity separation using instance labels or bounding boxes.
• Includes mesh baking for texture auto-completion and specular map generation.
• Generates convex and bounding-mesh collision models.

Maintenance & Community

• Codebase builds on ngp-pl, instant-ngp-pp, nerf2mesh, and sketchbook.
• Contact: Hongchi Xia.

Licensing & Compatibility

• License not explicitly stated in the README.
• Compatible with Three.js and Unreal Engine for game development.

Limitations & Caveats

• The project is described as "Progress [ ] Baking Finetune Module", indicating incomplete features.
• Physical simulation in Three.js requires significant manual effort due to Cannon.js limitations; Unreal Engine is recommended for more complex interactions.