This repository provides VRAG-RL, a framework for multi-turn, multi-modal agent training designed to enhance visually rich information understanding through reinforcement learning. It targets researchers and developers working with vision-language models (VLMs) and retrieval-augmented generation (RAG) systems, offering a novel approach to iterative reasoning and information gathering.

How It Works

VRAG-RL employs a reinforcement learning framework to train VLMs for effective reasoning and retrieval in visually rich contexts. It enables agents to progressively gather information from coarse to fine-grained perspectives, utilizing a multi-turn, multimodal training approach with strong extensibility for various tools. The system integrates state-of-the-art visual embedding models for custom retriever creation.

Quick Start & Requirements

• Installation: Clone the repository and install dependencies using pip install -r requirements.txt.
• Prerequisites: Python 3.10, a search engine deployed on a single GPU (preferably A100 80G for a 7B model), and vLLM for model serving.
• Demo: Follow run_demo.sh for a step-by-step guide, involving deploying a search engine server, serving a VLM (e.g., Qwen2.5-VL-7B-VRAG) via vLLM, and launching the Streamlit demo.
• Custom Corpus: Convert documents to JPG, ingest them using search_engine/ingestion.py (built on Llama-Index), and run the search engine API server.
• Links: Demo, Framework, ViDoRAG.

Highlighted Details

• Offers a customizable VRAG demo.
• Integrates SOTA visual embedding models for custom retrievers.
• Releases a 7B model with plans for more sizes.
• Training code for VRAG-RL is under development.

Maintenance & Community

The project acknowledges contributions from ViDoRAG, LLaMA-Factory, Search-R1, and verl. Further research is available via ViDoRAG.

Licensing & Compatibility

The repository does not explicitly state a license in the README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The project is under ongoing development, with training code to be released soon. The README does not specify the license, which may impact commercial use.