Summary

This repository provides a framework for evaluating end-to-end visual document retrieval pipelines on the ViDoRe v3 benchmark datasets. It targets researchers and practitioners seeking to assess complex retrieval systems, offering a centralized platform for submitting and comparing results, thereby accelerating progress in the field.

How It Works

The framework enables evaluation of complete retrieval systems, including multi-stage (e.g., retrieve-rerank), hybrid (e.g., dense-sparse fusion), and custom preprocessing pipelines. It moves beyond traditional single-component evaluation by allowing arbitrary retrieval logic. The system leverages the ViDoRe v3 benchmark datasets and serves as a community results repository, encouraging submissions of pipeline evaluations and descriptions.

Quick Start & Requirements

Installation is straightforward via pip: pip install vidore-benchmark. The framework supports multiple ViDoRe v3 datasets, including vidore/vidore_v3_hr, vidore/vidore_v3_finance_en, and vidore/vidore_v3_industrial. Users can list available datasets with vidore-benchmark pipeline list-datasets and evaluate custom pipelines using vidore-benchmark pipeline evaluate. A Python API is also available for programmatic evaluation. Specific hardware or software prerequisites beyond standard Python are not detailed.

Highlighted Details

• Facilitates evaluation of complex, multi-stage, and hybrid retrieval architectures.
• Acts as a community hub for submitting pipeline results and descriptions via Pull Requests.
• Tracks core retrieval metrics alongside indexing and search computing times.
• Supports optional detailed tracking of additional metrics like estimated cost, GPU usage, and timing.
• Maintains reproducibility for legacy benchmarks (v1/v2) and vision retriever evaluations, though these are deprecated.

Maintenance & Community

The project's active development focus has shifted to pipeline evaluation for ViDoRe v3. Older functionalities, including vision retriever evaluation and benchmarks v1/v2, are deprecated and no longer actively maintained. No explicit community channels or contributor details are provided in the README.

Licensing & Compatibility

The project is released under the MIT License, which is permissive for commercial use and integration into closed-source projects.

Limitations & Caveats

The repository explicitly deprecates and ceases active maintenance for its prior focus areas (vision retriever evaluation, ViDoRe v1/v2 benchmarks), directing users to MTEB for those tasks. Detailed setup times, resource footprints, and specific hardware requirements (e.g., GPU) are not provided within the README.