<2-3 sentences summarising what the project addresses and solves, the target audience, and the benefit.> VLM³ introduces a paradigm shift in 3D vision, demonstrating that standard Vision-Language Models (VLMs) are native 3D learners. It enables researchers and practitioners to achieve state-of-the-art results on various 3D understanding tasks without resorting to complex, task-specific architectures, losses, or data augmentations. The primary benefit is a simplified, scalable approach to 3D learning, relying on generalist foundation models and data scaling.

How It Works

<2-4 sentences on core approach / design (key algorithms, models, data flow, or architectural choices) and why this approach is advantageous or novel.> The core innovation lies in adapting standard VLMs for 3D tasks through simple preprocessing and a unified text-based interface. Input images are resized to normalize focal length, resolving camera ambiguity without extra encoders. 3D points or pixels are referenced using normalized text coordinates (e.g.,), eliminating the need for architectural modifications or marker rendering. Training utilizes standard VLM architectures and supervised fine-tuning (SFT), highlighting that large models, task-specific designs, and complex formulations are unnecessary for effective 3D learning when combined with data scaling.

Quick Start & Requirements

• Primary install / run command (pip, Docker, binary, etc.). pip install transformers>=5.4.0
• Non-default prerequisites and dependencies (GPU, CUDA >= 12, Python 3.12, large dataset, API keys, OS, hardware, etc.). Requires the transformers library. Inference utilizes base VLM architectures (e.g., Qwen3-vl-4B).
• Estimated setup time or resource footprint. Not specified.
• If they are present, include links to official quick-start, docs, demo, or other relevant pages. A Python code snippet demonstrates chat-based inference for depth estimation using the facebook/VLM3-depth checkpoint. A cookbook for detailed examples is mentioned in the README.

Highlighted Details

• Surpasses SpatialRGPT on object-level 3D understanding without additional encoders.
• Matches UnidepthV2 and Moge-2 on metric depth estimation, improving DepthLM accuracy from 0.84 to 0.9.
• Outperforms DKM and RoMa for pixel correspondence estimation.
• Matches DepthAnything3 and surpasses VGGT for camera pose estimation.

Maintenance & Community

• Notable contributors, sponsorships, partnerships, deprecations, migrations, or other health signals if notable. Contact: Zhipeng Cai (Meta Inc).
• Links to Discord/Slack, social handles, roadmap, etc. Homepage: https://zhipengcai.github.io/, email: czptc2h at gmail dot com. No explicit community channels (e.g., Discord, Slack) are listed.

Licensing & Compatibility

• License type and notable restrictions (GPL -> copyleft, SSPL, etc.). FAIR CC-BY-NC (Non-Commercial).
• Compatibility notes for commercial use or closed-source linking. The non-commercial restriction prohibits integration into proprietary or revenue-generating applications.

Limitations & Caveats

<1-3 sentences on caveats: unsupported platforms, missing features, alpha status, known bugs, breaking changes, bus factor, deprecation, etc. Avoid vague non-statements and judgments.> Models are currently listed as "Coming Soon!". The FAIR CC-BY-NC license strictly limits usage to non-commercial research purposes. Performance claims are based on specific benchmarks and may vary in real-world applications.