This project provides RadFM, a generalist foundation model for radiology, capable of processing both 2D and 3D medical scans with interleaved visual and language inputs. It is designed for researchers and practitioners in medical AI who need a versatile model for various radiological tasks.

How It Works

RadFM leverages a multi-modal generative approach, integrating visual tokens from a 3D Vision Transformer and Perceiver directly into a LLaMA-based causal language model. This allows for seamless fusion of image and text data, enabling tasks like image captioning and diagnosis. A custom My_trainer and datasampler.py are used to ensure that 2D and 3D data are not mixed within the same batch, preventing computational overhead from data expansion and improving training efficiency.

Quick Start & Requirements

• Install/Run: Download model checkpoint, decompress, place pytorch_model.bin in Quick_demo/, then run python test.py.
• Prerequisites: Requires at least one Nvidia A100 (80GB) GPU for acceptable inference performance.
• Resources: Significant GPU memory is needed.
• Links: Quick Demo, Datasets

Highlighted Details

• Trained on a large-scale MedMD dataset comprising 16 million 2D and 3D medical images.
• Supports multi-image input and visual-language interleaving.
• Custom trainer prevents mixing 2D/3D data in batches for efficiency.
• Utilizes a Perceiver and 3D ViT for image tokenization.

Maintenance & Community

• Contact: wtzxxxwcy02@sjtu.edu.cn
• Datasets are released and available on Hugging Face.

Licensing & Compatibility

• The README does not explicitly state a license. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

• Inference requires a high-end GPU (Nvidia A100 80GB) to avoid extremely slow performance.
• The key_words_query functionality mentioned in the embedding layer is currently unused.