Summary

LongVT addresses LMM hallucinations in long-form video analysis by enabling "Thinking with Long Videos" via native tool calling. It employs a global-to-local reasoning loop, using LMM temporal grounding as a video cropping tool to dynamically resample frames and ground answers in visual evidence. This enhances LMM accuracy and reliability on complex video understanding tasks for researchers and practitioners.

How It Works

The framework mimics human video comprehension: global skimming followed by local clip examination. It leverages LMM temporal grounding as a native video cropping tool to dynamically zoom into specific segments and resample frames. This iterative global-to-local reasoning loop continues until answers are visually grounded. Training uses the VideoSIAH dataset (~248K SFT, ~1.6K RL, ~15K RFT samples) via a three-stage strategy.

Quick Start & Requirements

Installation varies: SFT/RFT use lmms-engine; RL requires cloning, Conda env (python=3.10), and scripts/install_vllm_sglang_mcore.sh. Data pipeline uses uv pip install -e . or pip install -e .. Key dependencies include decord, scenedetect, vllm, sglang, and ray (for distributed RL). GPU acceleration is essential. Links to paper, code, data, and models are provided.

Highlighted Details

• Accepted to CVPR 2026.
• Received AI Paper of the Day and Top Weekly/Monthly Paper awards (late 2025).
• Features a native video cropping tool for dynamic global-to-local visual evidence retrieval.
• Employs an XML-style schema for tool calls, improving parseability for smaller models over JSON.
• Trained on the extensive VideoSIAH dataset (~248K SFT, ~1.6K RL, ~15K RFT samples).

Maintenance & Community

The project welcomes community contributions, particularly to the verl integration. The GitHub repository serves as the primary interaction hub.

Licensing & Compatibility

License information is not explicitly stated in the README, requiring further investigation for commercial use or integration.

Limitations & Caveats

Format fragility can occur with parallel, multi-tool calls (addressed in ParaVT). Evaluation requires specific vLLM versions (0.12.0), chat templates, and high max_new_tokens (49152) to prevent truncation. LLM Judge misconfiguration leads to inaccurate scores. crop_video uses absolute timestamps (seconds).