This repository provides tools and pre-trained models for Dense Passage Retrieval (DPR), a state-of-the-art approach for open-domain question answering. It's designed for researchers and practitioners in NLP and information retrieval looking to implement or experiment with dense retrieval systems. The primary benefit is enabling efficient and accurate retrieval of relevant passages for answering questions from large text corpora.

How It Works

DPR utilizes a bi-encoder architecture where separate BERT-based encoders process questions and passages. These encoders generate dense vector representations, and retrieval is performed by finding passages whose vectors are closest (via dot product) to the question vector. This approach allows for efficient similarity search using FAISS, outperforming traditional sparse methods like BM25 in many benchmarks.

Quick Start & Requirements

• Install via pip: pip install .
• Requires Python 3.6+ and PyTorch 1.2.0+.
• Supports Huggingface (<=3.1.0), Pytext, and Fairseq encoders.
• Pre-trained models and datasets can be downloaded using python data/download_data.py.

Highlighted Details

• State-of-the-art performance on NQ dataset, with a new model achieving 52.47% top-1 passage retrieval accuracy.
• Hydra-based configuration for command-line tools.
• Pluggable data processing layer for custom datasets.
• Inference utilizes FAISS for efficient indexing and retrieval.

Maintenance & Community

This project is from Facebook AI Research. Further community engagement details are not specified in the README.

Licensing & Compatibility

• License: CC-BY-NC 4.0 (Creative Commons Attribution-NonCommercial 4.0 International).
• Non-commercial use restriction: This license prohibits commercial use.

Limitations & Caveats

The CC-BY-NC 4.0 license restricts commercial applications. WebQ validation requires entity normalization, which is not currently implemented.