MedSAM3-v1 addresses the need for flexible, text-guided medical image segmentation by incorporating specific medical concepts. It targets researchers and practitioners in medical imaging, offering a benefit of concept-driven segmentation across diverse modalities without relying on traditional point or box prompts.

How It Works

This project fine-tunes the SAM3 model using a parameter-efficient LoRA (Low-Rank Adaptation) strategy. The core innovation lies in its pure text-guided approach, enabling segmentation based on medical concepts rather than geometric inputs. This method allows for robust segmentation across a wide array of medical imaging types by leveraging a large-scale dataset annotated with numerous medical text IDs.

Quick Start & Requirements

• Installation: Clone the repository (git clone https://github.com/Joey-S.Liu/MedSAM3.git), navigate into the directory (cd MedSAM3), install dependencies (pip install -e .), and authenticate with Hugging Face (hf auth login).
• Prerequisites: Hugging Face account and authentication token.
• Inference: Execute python3 infer_sam.py --config configs/full_lora_config.yaml --image path/to/image.jpg --prompt "skin lesion" --threshold 0.5 --nms-iou 0.5 --output skin_lesion.png.
• Training: Run python3 train_sam3_lora_native.py --config configs/full_lora_config.yaml.
• References: SAM3 (https://github.com/facebookresearch/sam3), SAM3_LoRA (https://github.com/Sompote/SAM3_LoRA), Hugging Face Weights (https://huggingface.co/lal-Joey/MedSAM3_v1), arXiv (https://arxiv.org/abs/2511.19046).

Highlighted Details

• Dataset Scale: Features 658,094 images, 2,863,974 instance annotations, and 330 unique medical text IDs (concepts).
• Modalities: Supports Radiology (CT, MRI, PET, X-ray), Optical/Microscopic (Microscopy, Histopathology, Dermoscopy, OCT, Cell), and Video/Procedure (Ultrasound, Endoscopy, Surgery video).
• Methodology: Employs LoRA fine-tuning on SAM3 for parameter efficiency.

Maintenance & Community

The project is continuously updated. Contact information for corresponding authors is provided: Xu R. Cao (xucao2@illinois.edu) and Jintai Chen (jintaiCHEN@hkust-gz.edu.cn).

Licensing & Compatibility

No license information is explicitly stated in the provided README. This absence may pose compatibility concerns for commercial use or integration into closed-source projects.

Limitations & Caveats

The full list of supported task categories for v1 is pending release. Users are encouraged to experiment with specific tasks and provide feedback. Optimal segmentation performance may require flexible adjustment of hyperparameters like threshold and nms-iou based on the modality and segmentation target.