fMRI-reconstruction-NSD  by MedARC-AI

fMRI-to-image reconstruction & retrieval on the NSD dataset

created 2 years ago
333 stars

Top 83.6% on sourcepulse

GitHubView on GitHub
1 Expert Loves This Project
Project Summary

This repository provides tools for reconstructing and retrieving images from fMRI data using deep learning models, specifically targeting researchers and practitioners in cognitive neuroscience and AI. It enables the translation of brain activity patterns into visual representations, facilitating a deeper understanding of neural encoding of visual information.

How It Works

The project leverages contrastive learning and diffusion models to map fMRI data to image representations. It utilizes CLIP embeddings (from last hidden layer or final layer) and the variational autoencoder of Stable Diffusion. The approach allows for both direct image reconstruction and retrieval from large datasets like LAION-5B, offering flexibility in how brain activity is translated into visual content.

Quick Start & Requirements

Highlighted Details

  • Supports fMRI-to-image reconstruction and retrieval.
  • Integrates with LAION-5B for large-scale image retrieval.
  • Offers pre-trained models for Subject 1 of NSD.
  • Includes scripts for training, reconstruction, retrieval, and evaluation.

Maintenance & Community

The project is associated with the MedARC-AI group. A newer version, MindEye2, is available, which reportedly outperforms MindEye1.

Licensing & Compatibility

The repository's license is not explicitly stated in the README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The project requires significant data downloads (NSD) and computational resources for training. Pretrained models are only available for Subject 1 of the NSD dataset.

Health Check
Last commit

1 year ago

Responsiveness

1 day

Pull Requests (30d)
0
Issues (30d)
0
Star History
12 stars in the last 90 days

Explore Similar Projects

Feedback? Help us improve.