Summary

Holi-Spatial provides a data curation pipeline to transform video streams into holistic 3D spatial intelligence. It enables researchers and developers to generate 3D Gaussian Splatting models, meshes, object annotations, and spatial QA datasets from existing scene datasets like ScanNet. The project aims to evolve video data into rich, actionable 3D scene representations.

How It Works

The pipeline operates in three stages: geometric optimization using DA3 and 3DGS training; image-level perception via VLMs for discovery and SAM3 for mask generation; and scene-level refinement, lifting 2D masks to 3D bounding boxes, and synthesizing spatial QA. This multi-stage approach integrates state-of-the-art foundation models to derive comprehensive 3D scene understanding.

Quick Start & Requirements

• Installation: Requires a CUDA-enabled Python environment. Install common dependencies with pip install -r requirements.txt, followed by pip install -e PGSR/submodules/diff-plane-rasterization and pip install -e PGSR/submodules/simple-knn.
• Prerequisites: A VLM served through an OpenAI-compatible vLLM endpoint (defaulting to http://localhost:8000/v1). SAM3 checkpoint is downloaded automatically.
• Data: Expects specific directory structures for ScanNet v2, ScanNet++, and DL3DV datasets, including depth, point clouds, and covisibility information.
• Links: Paper: https://arxiv.org/abs/2603.07660.

Highlighted Details

• Featured as an ICML 2026 Oral paper.
• Releases a subset of the Holi-Spatial dataset: 2,000+ Gaussian models, meshes, 3D bounding boxes, and the 2-million-scale HoliSpatial-QA-2M dataset.
• The pipeline generates 3DGS geometry, mesh-guided masks, object/region 3D annotations, spatial QA, and LLaMA-Factory training data.

Maintenance & Community

The repository is associated with an ICML 2026 paper and builds upon several open research works and foundation models. Specific community links or active maintenance signals beyond the paper's release are not detailed in the README.

Licensing & Compatibility

No explicit software license is mentioned in the provided README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The pipeline code is slated for release in July, suggesting it may be in a pre-release or beta state. The setup requires significant computational resources, including a CUDA-enabled environment and a running vLLM server. Input data must conform to strict directory layouts for ScanNet, ScanNet++, and DL3DV.