MedRAX is a versatile AI agent designed for comprehensive Chest X-ray (CXR) interpretation, addressing the limitations of isolated CXR analysis tools. It integrates state-of-the-art CXR analysis models with multimodal LLMs to answer complex medical queries without retraining, targeting researchers and clinicians aiming for practical automated CXR interpretation.

How It Works

MedRAX is built on LangChain and LangGraph, utilizing GPT-4o as its backbone LLM. It dynamically orchestrates a suite of specialized tools for visual QA (CheXagent, LLaVA-Med), segmentation (MedSAM, PSPNet), grounding (Maira-2), report generation (SwinV2), and disease classification (DenseNet-121). This modular, tool-agnostic architecture allows for flexible integration of new capabilities and dynamic query resolution.

Quick Start & Requirements

• Install: pip install -e .
• Run: python main.py
• Prerequisites: Python 3.8+, CUDA/GPU recommended. OpenAI API key required for GPT-4o. Manual setup for RoentGen weights.
• Setup: Requires cloning the repo, installing dependencies, setting model_dir and OPENAI_API_KEY.
• Docs: MedRAX Docs (Note: Link points to system prompts, not a dedicated docs page).

Highlighted Details

• Integrates 7 specialized CXR analysis tools: Visual QA, Segmentation, Grounding, Report Generation, Disease Classification, X-ray Generation, and Utilities.
• Introduces ChestAgentBench, a benchmark with 2,500 complex medical queries across 7 categories for rigorous evaluation.
• Achieves state-of-the-art performance compared to open-source and proprietary models.
• Supports local and cloud deployments with a production-ready Gradio interface.

Maintenance & Community

• Developed by researchers from the University of Toronto, Vector Institute, and University Health Network.
• Citation details provided for the associated ICML 2025 paper.

Licensing & Compatibility

• The repository itself is not explicitly licensed in the README. The associated paper is available on arXiv.

Limitations & Caveats

The current experimental release does not support vision for GPT-4o and MedSAM, with these features planned for integration. Manual setup is required for RoentGen weights. Some tools are resource-intensive, and selective tool initialization is recommended for memory management.