fMRI-to-image reconstruction & retrieval on the NSD dataset
Top 83.6% on sourcepulse
This repository provides tools for reconstructing and retrieving images from fMRI data using deep learning models, specifically targeting researchers and practitioners in cognitive neuroscience and AI. It enables the translation of brain activity patterns into visual representations, facilitating a deeper understanding of neural encoding of visual information.
How It Works
The project leverages contrastive learning and diffusion models to map fMRI data to image representations. It utilizes CLIP embeddings (from last hidden layer or final layer) and the variational autoencoder of Stable Diffusion. The approach allows for both direct image reconstruction and retrieval from large datasets like LAION-5B, offering flexibility in how brain activity is translated into visual content.
Quick Start & Requirements
conda activate mindeye
after running setup.sh
.Highlighted Details
Maintenance & Community
The project is associated with the MedARC-AI group. A newer version, MindEye2, is available, which reportedly outperforms MindEye1.
Licensing & Compatibility
The repository's license is not explicitly stated in the README. Compatibility for commercial use or closed-source linking is not specified.
Limitations & Caveats
The project requires significant data downloads (NSD) and computational resources for training. Pretrained models are only available for Subject 1 of the NSD dataset.
1 year ago
1 day