NeoVerse enhances 4D world models from in-the-wild monocular videos, enabling 4D reconstruction and novel-trajectory video generation. It targets researchers and developers, offering a versatile tool for advanced 3D scene understanding and video synthesis from single-source inputs.

How It Works

It integrates a 3D reconstructor (e.g., WorldMirror, Depth Anything 3) to derive Gaussian Splats and camera poses from monocular video. This 4D scene representation feeds a video diffusion model (WAN 2.1 backbone with distilled LoRA) for high-quality frame synthesis. The modular design supports interchangeable reconstructors and fast inference, conditioned on prompts and user-defined trajectories.

Quick Start & Requirements

Installation involves cloning, setting up a Python 3.10 conda environment, and installing specific PyTorch versions (e.g., 2.3.1 for CUDA 12.1 or 2.7.1 for CUDA 12.8) and dependencies (torch-scatter, gsplat). Model checkpoints are downloadable via Hugging Face or ModelScope. Usage is via command-line (inference.py) or Gradio demo (app.py). Tested on CUDA 12.1/PyTorch 2.3.1 and CUDA 12.8/PyTorch 2.7.1.

Highlighted Details

• Fast inference (<30s on A800 with distilled LoRA).
• Plug-and-play integration of alternative 3D reconstructors (e.g., Depth Anything 3).
• Flexible camera trajectory control: predefined motions or custom JSON trajectories.
• Interactive Gradio web UI for simplified usage.

Maintenance & Community

Recent updates include inference scripts and model checkpoints (Feb 2026), and the arXiv paper (Jan 2026). The project acknowledges several open-source inspirations. Contact is via email, WeChat, or GitHub issues for support and feedback.

Licensing & Compatibility

The README omits explicit license information, posing a significant adoption blocker, especially for commercial use, as licensing terms remain undefined.

Limitations & Caveats

Installation instructions specify tested CUDA/PyTorch versions, indicating potential compatibility issues. As a recent project (checkpoints Feb 2026), long-term maintenance and community adoption are developing. The lack of licensing details is a critical caveat.