Open & Reproducible Research for Tracking VLAs (OmTrackVLA) addresses the challenge of visual navigation and following for embodied AI systems. It provides a fully open-source Vision-Language-Action (VLA) stack that translates monocular video and natural-language instructions into actionable, short-horizon waypoints. This project targets researchers and developers seeking to democratize embodied AI, offering an accessible platform for reproduction, fine-tuning, and deployment with affordable compute.

How It Works

OmTrackVLA implements a VLA stack that processes video history and text instructions to predict future waypoints for robot control. Its core approach involves a Qwen-based planner combined with masked waypoint losses. To enhance efficiency, it precomputes vision tokens (DINO/SiGLIP embeddings) allowing the trainer to memory-map tensors instead of re-encoding frames on the fly. This design prioritizes accessibility and reproducibility, building upon the foundation of the original TrackVLA project.

Quick Start & Requirements

• Install: Create a Conda environment (python=3.9, cmake=3.14.0), activate it, install habitat-sim==0.3.1 (with Bullet), clone the repository, and pip install -e habitat-lab.
• Data: Requires HM3D and MP3D scene datasets (request access via Habitat links) and humanoid avatar data (download via script or manual zip).
• Processing: Use make_tracking_data.py to generate training shards from raw data and precache_frames.py to cache vision embeddings.
• Prerequisites: Python 3.9, Conda, Habitat-Sim, Habitat-Lab. Substantial dataset downloads are necessary.
• Links: Repository, 0.6B Checkpoint.

Highlighted Details

• Performance: The latest 0.6B checkpoint achieves a Success Rate (SR) of 60.04 in the AT setting, surpassing the 7B TrackVLA baseline (50.2), and leads on STT/AT Tracking Rate (TR).
• Openness: Releases model weights, inference code, and the full training pipeline, fostering community-driven improvements.
• Efficiency: Utilizes a highly efficient 0.6B parameter model and precomputes vision tokens to accelerate training.
• Data Release: The full training data is planned for future release.

Maintenance & Community

The project is actively maintained with updated model weights and benchmark scores. Contributions and issue reports are welcomed via pull requests. The full training data is slated for future release.

Licensing & Compatibility

The project is described as "fully open-source" and aims to democratize embodied AI. However, the specific license type and any associated restrictions for commercial use or closed-source linking are not explicitly stated in the README. Users must verify licensing details.

Limitations & Caveats

Collision rates in denser scenarios (STT, DT) remain higher than the TrackVLA baseline and are an active area for improvement. The full training dataset is not yet publicly available.