Rein provides an efficient and robust fine-tuning method for Domain Generalized Semantic Segmentation (DGSS) using Vision Foundation Models (VFMs). It is designed for researchers and practitioners aiming to achieve state-of-the-art performance on unseen domains with minimal data.

How It Works

Rein employs a parameter-efficient fine-tuning strategy, focusing on adapting VFMs to new segmentation tasks without retraining the entire model. This approach leverages the strong feature extraction capabilities of VFMs, enabling high performance with significantly reduced computational cost and data requirements. The method has demonstrated SOTA results on challenging cross-domain segmentation benchmarks.

Quick Start & Requirements

• Install: Clone the repository and set up a Conda environment with PyTorch 2.0.1, CUDA 11.7, and MMSegmentation.
• Prerequisites: Python 3.8+, PyTorch 2.0.1, CUDA 11.7, MMSegmentation, MMDetection, MMCV, OpenMim, xformers (optional).
• Data: Requires downloading and converting datasets like Cityscapes, Mapillary, GTA, and optionally ACDC and UrbanSyn.
• Weights: Pre-trained VFM weights (e.g., DINOv2) are required and need conversion.
• Demo: A demo.ipynb notebook is available for exploration.
• Docs: Detailed instructions for setup, data preparation, training, and evaluation are provided.

Highlighted Details

• Achieves 78.4% mIoU on Cityscapes using only synthetic data (UrbanSyn).
• Achieves 77.6% mIoU on ACDC using only Cityscapes training data.
• Supports various backbones including ResNet, ConvNeXt, ViT, CLIP, SAM, and EVA02.
• Parameter-efficient fine-tuning reduces storage and computational overhead.

Maintenance & Community

The project is associated with CVPR 2024 and lists authors from the University of Science and Technology of China and Shanghai AI Laboratory. A simplified version (simple_reins) is available for easier integration.

Licensing & Compatibility

The repository does not explicitly state a license in the README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The project relies heavily on specific versions of PyTorch and CUDA, and requires significant data preparation. The README mentions that the simple_reins version has removed features related to Mask2Former, implying the main repository uses Mask2Former.