This repository provides MST++, a multi-stage spectral-wise Transformer for efficient spectral reconstruction, and serves as a toolbox for 11 image restoration algorithms in this domain. It is targeted at researchers and practitioners in computer vision and hyperspectral imaging, offering state-of-the-art performance and a comprehensive benchmark.

How It Works

MST++ utilizes a novel Spectral-wise Multi-head Self-attention (S-MSA) mechanism, designed to leverage the spectral self-similarity of hyperspectral images. This forms the basis of Spectral-wise Attention Blocks (SABs), which are then integrated into a U-shaped architecture (Single-stage Spectral-wise Transformer, SST) for multi-resolution feature extraction. Multiple SSTs are cascaded to progressively refine the spectral reconstruction quality from coarse to fine, outperforming CNN-based methods by better capturing long-range dependencies.

Quick Start & Requirements

• Install: Clone the repository and run pip install -r requirements.txt.
• Prerequisites: Python 3 (Anaconda recommended), NVIDIA GPU with CUDA.
• Data: Download datasets for training, validation, and testing from provided links and organize them as specified.
• Pre-trained Models: Download from provided links for evaluation and prediction.
• Links: CVPRW 2022 Paper, NTIRE 2022 Spectral Recovery Challenge, Awesome-Transformer-Attention

Highlighted Details

• Winner of the NTIRE 2022 Spectral Reconstruction Challenge.
• Achieves state-of-the-art performance with significantly fewer parameters and FLOPS compared to other methods.
• Includes implementations and benchmarks for 11 different spectral reconstruction algorithms.
• Offers functions for evaluating model parameters and FLOPS.

Maintenance & Community

The project is associated with CVPRW 2022 and NTIRE 2022. Recent updates indicate ongoing work on related methods like Retinexformer, which ranked highly in NTIRE 2024 challenges.

Licensing & Compatibility

The repository does not explicitly state a license. However, the inclusion of academic paper citations suggests it is intended for research purposes. Commercial use may require clarification.

Limitations & Caveats

The setup requires downloading multiple datasets and pre-trained models from external links, which can be time-consuming. The project is primarily focused on spectral reconstruction from RGB images, and its applicability to other domains or data types is not specified.