OpenSearch-VL offers a comprehensive, open-source recipe for training advanced multimodal deep search agents. It addresses the challenge of reproducing state-of-the-art agents by releasing curated datasets, training code, and model checkpoints. This project is targeted at researchers and engineers seeking to develop or leverage powerful agents capable of complex visual understanding and web-scale information retrieval, providing a significant benefit by democratizing access to frontier multimodal search capabilities.

How It Works

The project employs a multi-stage approach. First, a data curation pipeline synthesizes image-grounded, multi-hop Visual Question Answering (VQA) instances from the Wikipedia hyperlink graph, utilizing fuzzy entity rewriting and source-anchored visual grounding to create the SearchVL-SFT-36k and SearchVL-RL-8k datasets. Second, a unified environment provides a diverse set of visual and retrieval tools (e.g., crop, text_search, web_search, perspective_correct) that agents use to interact with information. Finally, training utilizes a novel multi-turn fatal-aware GRPO algorithm, which robustly handles cascading tool failures by masking out tokens following such errors and employing one-sided advantage clamping to preserve valid reasoning, leading to improved performance and stability.

Quick Start & Requirements

• Primary Install/Run: Installation involves setting up separate virtual environments for SFT (code/SFT/) and RL (code/RL/) components, followed by using the opensearch_vl/ package for inference. Key commands include pip install -e for sub-projects and llamafactory-cli train for SFT, python -m vision_deepresearch_async_workflow.train_deepresearch_workflow_megatron for RL, and python opensearch_vl/run_infer.py for inference.
• Prerequisites: Python 3.10+, CUDA 12.1+ (12.4 recommended), PyTorch >= 2.4 with CUDA support. GPU requirements range from a single H100/A100-80GB for the 8B model to multi-node setups for 30B/32B models. Dependencies include DeepSpeed, Ray, Megatron-LM, and sglang. Optional API keys for services like Serper, Jina, and Qwen are supported.
• Links:
  • Paper: https://arxiv.org/pdf/2605.05185
  • GitHub: https://github.com/shawn0728/OpenSearch-VL
  • Hugging Face Collection: https://huggingface.co/OpenSearch-VL

Highlighted Details

• Provides fully open-source datasets (SearchVL-SFT-36k, SearchVL-RL-8k) and model checkpoints for frontier multimodal search agents.
• Features a unified tool environment supporting visual enhancement, parsing, and retrieval across SFT, RL, and inference.
• Achieves state-of-the-art results, with OpenSearch-VL-32B matching proprietary systems and OpenSearch-VL-8B outperforming existing open 8B agents.
• The fatal-aware GRPO algorithm significantly improves agent performance (+4.2 points over vanilla GRPO) by better handling tool failures.

Maintenance & Community

The project is actively developed, with a TODO list indicating planned releases for the data curation pipeline and a public demo. Community interaction appears primarily driven through GitHub and potentially Twitter announcements. Specific details on maintainers or community forums like Discord/Slack are not detailed in the README.

Licensing & Compatibility

Project-specific additions are released under the Apache 2.0 license, which is permissive for commercial use. However, the repository bundles several upstream open-source projects (e.g., LLaMA-Factory, rLLM, Megatron-LM), each retaining its original license. Users must ensure compatibility with these underlying licenses.

Limitations & Caveats

The project is presented with an "alphaXiv" badge and ongoing TODOs, suggesting it is in an active development phase. Training and running the larger models (30B, 32B) require substantial multi-node GPU infrastructure. The setup process involves integrating multiple complex libraries, demanding a high level of technical expertise. Functionality may depend on optional external API keys.