This repository provides the official codebase for the paper "Assessing and Enhancing Large Language Models in Rare Disease Question-answering." It offers a benchmark dataset, ReDis-QA, and a corpus, ReCOP, designed to evaluate and improve LLM performance on rare disease-related queries, targeting researchers and developers in medical AI.

How It Works

The project evaluates LLMs using a question-answering framework focused on rare diseases. It supports both zero-shot LLM execution and Retrieval-Augmented Generation (RAG) approaches. RAG implementations utilize metadata retrievers, BM25, and MedCPT retrievers, with options to combine the ReCOP corpus with baseline corpora like PubMed, Textbooks, and Wikipedia for enhanced retrieval accuracy.

Quick Start & Requirements

• Install: pip install -r requirements.txt
• Prerequisites: Python 3.10.0, PyTorch compatible with CUDA (e.g., 2.4.0+cu121), Git LFS (for initial corpus download), Java (for BM25 retriever).
• Dataset/Corpus Loading: Uses Hugging Face datasets library (load_dataset).
• Scripts: zero-shot-bench/scripts/run_exp.sh, meta-data-bench/scripts/run_exp.sh, rag-bench/scripts/run_exp.sh, combine-corpora-bench/scripts/run_exp.sh.
• Resources: Links to dataset and corpus loading examples are provided.

Highlighted Details

• ReDis-QA dataset covers 205 rare disease types, with questions categorized by symptoms, causes, effects, related disorders, and diagnosis.
• Evaluates LLMs in zero-shot and RAG configurations using various retrievers (metadata, BM25, MedCPT).
• Supports combining the ReCOP corpus with baseline corpora (PubMed, Textbook, Wikipedia, StatPearls).
• Baseline retrievers and corpora are sourced from the MedRAG open-source repository.

Maintenance & Community

• The project is associated with a research paper published on arXiv.
• No specific community channels (Discord, Slack) or active maintenance signals are mentioned in the README.

Licensing & Compatibility

• The README does not explicitly state a license.
• Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The project is presented as the codebase for a research paper, implying it may be experimental. The absence of a specified license and community support could pose adoption challenges.