Summary

This repository addresses knowledge-intensive Visual Question Answering (VQA) by introducing Retrieval Augmented VQA (RAVQA) and its advanced versions, RAVQA-v2 and PreFLMR. It enables models to retrieve relevant external knowledge for answering complex visual questions, benefiting researchers and engineers in multi-modal AI.

How It Works

The core innovation is the Fine-grained Late-interaction Multi-modal Retriever (FLMR) and its pre-trained variant, PreFLMR. FLMR facilitates detailed, late-stage interaction between visual and textual modalities for precise knowledge retrieval. PreFLMR, trained on extensive multi-modal data, acts as a robust foundation. The system integrates this retrieval with language models for answer generation, utilizing pytorch-lightning and Runway For ML for efficient experimentation, with FAISS and ColBERT powering retrieval.

Quick Start & Requirements

Installation requires a Python 3.8 environment with specific library versions (e.g., PyTorch 1.12.1+cu113, transformers 4.28.1, pytorch-lightning 2.0.4) and faiss-gpu. Feature extraction for components like VinVL may need separate environments and specific CUDA versions. Significant data downloads (COCO, OK-VQA) are necessary. ElasticSearch is recommended for feature indexing. Links to the ACL 2024 paper and Huggingface models are available.

Highlighted Details

• PreFLMR: A general-purpose multi-modal retriever pre-trained on >10 million data points, achieving state-of-the-art on knowledge-intensive tasks.
• RAVQA-v2: Features Fine-grained Late-interaction Multi-modal Retrieval for enhanced VQA.
• Benchmark Performance: Strong results across WIT, MSMARCO, OKVQA, and LLaVA benchmarks.
• Multilingual Support: Offers a released multilingual (Chinese + English) PreFLMR version.

Maintenance & Community

Active development is indicated by frequent news and releases, including ACL 2024 acceptance and a Huggingface implementation. Primary contributors are Weizhe Lin and Bill Byrne. No specific community channels or roadmap are detailed.

Licensing & Compatibility

Designated for "research purposes" with "all rights reserved," implying a restrictive, non-commercial license. Explicitly stated as "not a perfect framework for production."

Limitations & Caveats

The codebase was developed under time pressure, with plans for future readability/reproducibility improvements. Production deployment is not advised. Dynamic retrieval instructions are incomplete. Dependencies require specific, potentially older, library and CUDA versions.