CONCH is a vision-language foundation model for computational pathology, designed to advance AI in histopathology by integrating image and text data. It offers researchers and practitioners a powerful tool for a wide range of downstream tasks, including classification, segmentation, and retrieval, with the potential to reduce the need for extensive task-specific fine-tuning.

How It Works

CONCH employs contrastive learning from captions, pretraining on a large dataset of 1.17 million histopathology image-caption pairs. This approach allows it to learn rich representations from both visual and textual information, distinguishing it from models trained solely on images. This dual-modality learning enables better performance on non-H&E stained images and tasks requiring cross-modal understanding.

Quick Start & Requirements

• Install via pip install -e . after cloning the repository and activating a Python 3.10 conda environment.
• Requires PyTorch.
• Model weights must be downloaded separately from Hugging Face after requesting access.
• Detailed examples for loading and using the model are available in linked notebooks.

Highlighted Details

• State-of-the-art performance on 14 diverse computational pathology benchmarks.
• Trained on a unique dataset of 1.17M image-caption pairs, avoiding common public pathology datasets to prevent benchmark contamination.
• Capable of zero-shot classification, image-text retrieval, and serving as a general-purpose encoder for histopathology images and text.
• Benchmarked against multiple leading vision and vision-language models on various slide and ROI tasks.

Maintenance & Community

The project is associated with the Mahmood Lab. Recent updates include comparisons with other models like Virchow and Prov-GigaPath, and the release of related models like TITAN. The README lists numerous research applications and publications utilizing CONCH.

Licensing & Compatibility

Released under CC-BY-NC-ND 4.0 license. This strictly prohibits commercial use, sale, or monetization. Use is restricted to non-commercial, academic research purposes with proper attribution. Downloading requires Hugging Face registration and agreement to terms, including not distributing or reproducing the model.

Limitations & Caveats

The publicly released weights exclude the multimodal decoder due to potential PHI leakage concerns, though this does not affect the vision encoder's performance on key tasks. Commercial use is prohibited without prior approval.