Summary

MetaSpatial enhances 3D spatial reasoning in Vision-Language Models (VLMs) for realistic scene generation, targeting metaverse and AR/VR applications. It employs Reinforcement Learning (RL) to internalize spatial understanding, enabling adaptive and structured 3D scene creation in environments lacking perfect ground truth, offering a robust alternative to traditional supervised methods.

How It Works

The framework integrates RL to train VLMs (Qwen2.5 models) using physics-aware constraints and rendering image evaluations. This approach allows learning through adaptive rewards in imperfect environments, improving layout structure, physical consistency, and contextual realism compared to pre-training states.

Quick Start & Requirements

Setup is complex, requiring multiple Python environments (3.9, 3.11) and specific versions of PyTorch (1.12.1 with CUDA 11.3), MinkowskiEngine, DGL, and Blender (4.3.2). It necessitates installing libraries like transformers, objaverse, accelerate, OpenShape, and EasyR1. A detailed data curation pipeline involves OpenAI API calls and batch script generation for scene data. Links to arXiv, GitHub, Hugging Face datasets, IDesign, and EasyR1 are provided.

Highlighted Details

• Significantly improves Qwen2.5 VLMs (7B and 3B), with the 7B model showing greater gains in response stability and structure.
• RL training transforms disorganized object placements into structured, realistic, and contextually coherent 3D layouts.
• Designed for environments like the metaverse where precise ground truth is unavailable.

Maintenance & Community

Developed by Zhenyu Pan and Han Liu, building on VERL, EasyR1, RAGEN, and Zhuo Liu. No dedicated community channels are listed.

Licensing & Compatibility

The README does not specify a software license, requiring clarification for adoption, especially concerning commercial use or integration into closed-source projects.

Limitations & Caveats

The 3B Qwen2.5 model struggles with output format consistency. Current implementation supports only single-turn rollouts; multi-turn reasoning is planned. Hyperparameter tuning is incomplete, and training speed optimizations are suggested, including exploring faster rendering alternatives and local VLM scoring.