This repository provides tools for reconstructing and retrieving images from fMRI data using deep learning models, specifically targeting researchers and practitioners in cognitive neuroscience and AI. It enables the translation of brain activity patterns into visual representations, facilitating a deeper understanding of neural encoding of visual information.

How It Works

The project leverages contrastive learning and diffusion models to map fMRI data to image representations. It utilizes CLIP embeddings (from last hidden layer or final layer) and the variational autoencoder of Stable Diffusion. The approach allows for both direct image reconstruction and retrieval from large datasets like LAION-5B, offering flexibility in how brain activity is translated into visual content.

Quick Start & Requirements

• Install via conda activate mindeye after running setup.sh.
• Requires Natural Scenes Dataset (NSD) data (>30GB per subject).
• Optional: Download pretrained checkpoints for LAION-5B retrieval.
• Official Docs: https://medarc-ai.github.io/mindeye/
• arXiv: https://arxiv.org/abs/2305.18274

Highlighted Details

• Supports fMRI-to-image reconstruction and retrieval.
• Integrates with LAION-5B for large-scale image retrieval.
• Offers pre-trained models for Subject 1 of NSD.
• Includes scripts for training, reconstruction, retrieval, and evaluation.

Maintenance & Community

The project is associated with the MedARC-AI group. A newer version, MindEye2, is available, which reportedly outperforms MindEye1.

Licensing & Compatibility

The repository's license is not explicitly stated in the README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The project requires significant data downloads (NSD) and computational resources for training. Pretrained models are only available for Subject 1 of the NSD dataset.