MedAgents introduces a Multi-disciplinary Collaboration (MC) framework for zero-shot medical reasoning using Large Language Models (LLMs). It targets researchers and practitioners in medical AI, aiming to improve diagnostic accuracy and decision-making by simulating expert collaboration.

How It Works

The MC framework orchestrates LLM-based "experts" through five stages: gathering domain-specific experts, eliciting individual analyses, summarizing reports, facilitating collaborative consultation and iterative revision, and finally, reaching a consensus for decision-making. This approach leverages LLMs' ability to synthesize information and simulate expert dialogue for complex medical reasoning tasks.

Quick Start & Requirements

• Install dependencies: pip install -r requirements.txt
• Requires an OpenAI API key, to be placed in api_utils.py.
• Datasets are available via Google Drive: https://drive.google.com/file/d/11qNzDYIlimGGJ1fhQn2ux6w_rfFgJbyo/view?usp=sharing
• Inference script: sh inference.sh

Highlighted Details

• Evaluated on MedQA, MedMCQA, PubMedQA, and six MMLU medical subtasks.
• Focuses on zero-shot medical reasoning capabilities.
• Employs a multi-stage collaborative consultation process.

Maintenance & Community

The project is associated with the gersteinlab at Yale University and has an associated paper published in ACL 2024 Findings.

Licensing & Compatibility

The repository does not explicitly state a license. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The framework relies on OpenAI's API, potentially incurring costs and introducing vendor lock-in. The specific LLMs used and their performance characteristics are not detailed in the README.