This project provides Spann3R, a 3D reconstruction system that leverages spatial memory for improved accuracy and handling of dynamic scenes. It is targeted at researchers and developers in computer vision and robotics working on scene reconstruction from RGB-D data. The system offers enhanced reconstruction quality and supports both static and dynamic environments.

How It Works

Spann3R employs a novel spatial memory approach, building upon established techniques like NeRF and 3D Gaussian Splatting. It integrates a learned camera pose estimation module and a robust data preprocessing pipeline. The spatial memory allows the system to efficiently store and recall scene information, leading to more consistent and detailed reconstructions, particularly in dynamic scenarios where traditional methods might struggle.

Quick Start & Requirements

• Install: Clone the repository, create a conda environment (conda create -n spann3r python=3.9 cmake=1.14.0), install PyTorch with CUDA 11.8, and then pip install -r requirements.txt. Note: Open3D version 0.16.0 has a bug; use the development version. Compile CUDA kernels for RoPE.
• Prerequisites: Python 3.9, CMake 3.14.0, PyTorch 2.3.0 with CUDA 11.8, Open3D (dev version).
• Checkpoint: Download DUSt3R checkpoint.
• Demo: Download example data and run python demo.py.
• Nerfstudio Integration: Use --save_ori to save scaled intrinsics for training with ns-train splatfacto.
• Gradio Interface: Run python app.py for an interactive demo.
• Docs: data_preprocess.md

Highlighted Details

• Supports static and dynamic scene reconstruction.
• Achieves improved Chamfer distance metrics in v1.01 compared to v1.0.
• Integrates with Nerfstudio for further processing and rendering.
• Includes a Gradio interface for user-friendly interaction.

Maintenance & Community

The project is actively updated, with recent additions including support for Nerfstudio and camera parameter estimation. It acknowledges support from UKRI/EPSRC and Cisco Research.

Licensing & Compatibility

The repository does not explicitly state a license in the README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The README notes that Habitat sequences were removed from training due to performance issues on synthetic data. The Open3D installation requires using the development version to avoid a known bug.