XrayGLM is a Chinese multimodal large language model designed for chest radiograph summarization and interactive dialogue. It addresses the gap in medical multimodal research for the Chinese language, offering potential for medical image diagnosis and conversational AI in healthcare.

How It Works

XrayGLM is built by fine-tuning the VisualGLM-6B model on a custom Chinese chest X-ray dataset. This dataset was created by translating English reports from MIMIC-CXR and OpenI datasets using ChatGPT, aiming to improve accessibility and research for the Chinese medical community. The approach leverages existing powerful multimodal architectures and augments them with domain-specific, multilingual data.

Quick Start & Requirements

• Install: pip install -r requirements.txt (or requirements_wo_ds.txt to skip deepspeed).
• Prerequisites: Python, SwissArmyTransformer>=0.3.6.
• Inference: Run cli_demo.py or web_demo.py with specified model checkpoints.
• Training: Requires A100 GPUs (4x, 80GB recommended) and data preparation.
• Resources: Official video tutorials are available for fine-tuning and technical explanations.

Highlighted Details

• First Chinese medical multimodal model for chest X-ray summarization.
• Includes processed Chinese datasets (MIMIC-CXR-zh, OpenI-zh, OpenI-zh-plus).
• Demonstrates capabilities in image diagnosis and multi-turn dialogue.
• Offers fine-tuned weights for academic research.

Maintenance & Community

The project is a student-led initiative from the Macao Polytechnic University. Key contributors and advisors are listed. OpenAI credits were provided by USTC-PhD Yongle Luo for data translation.

Licensing & Compatibility

• License: CC BY-NC-SA.
• Restrictions: Non-commercial use is strictly enforced.

Limitations & Caveats

The provided checkpoints are labeled as "low quality." The model's output is not guaranteed for accuracy and should not be used for actual medical diagnosis. The project is intended strictly for academic research.