Research map / Vision-language and multimodal / Human and embodied modeling
NSD-Imagery: A Benchmark Dataset for Extending fMRI Vision Decoding Methods to Mental Imagery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Releases NSD-Imagery, a 7T fMRI benchmark of mental imagery paired with NSD, and benchmarks NSD-trained visual decoders (MindEye1/2, Brain Diffuser, iCNN, Takagi et al.) on cross-decoding to mental images.
Problem
Whether modern fMRI-to-image decoders trained on seen images in NSD generalize to mental imagery, which has lower SNR and spatial resolution, and whether complex natural-scene mental images can be reconstructed.
Approach
A new dataset extending NSD with mental imagery trials across simple, complex, and conceptual stimuli, used to cross-decode NSD-trained vision decoders, evaluated with image-feature metrics and large-scale human rater experiments.
Key results
- MindEye1 and Brain Diffuser performed best on mental imagery feature metrics (e.g., MindEye1 PixCorr 0.086, AlexNet(2) 59.56%), with Brain Diffuser best in generalization overall.
- Human raters (n=500) identified mental-imagery reconstructions at 73.00% (MindEye1) and 73.95% (Brain Diffuser) overall, while MindEye2 was near chance at 56.96%.
- Human similarity scores for matched vision and imagery reconstructions were correlated across stimuli (r=0.13 to 0.22, p<0.001), with MindEye2 showing the weakest correlation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a simple ridge-regression backbone with multimodal image and text features rather than the other tested architectures | NSD-trained vision decoders · NSD-Imagery mental imagery trials | human identification accuracy | Helped |
| Use MindEye2’s high-dimensional ViT-bigG embeddings and SDXL unCLIP generator instead of older tested decoder architectures | NSD-trained vision decoders · NSD-Imagery mental imagery trials | human identification accuracy: 83.05% -> 56.96% | Hurt |
| Use Takagi et al.’s +Decoded Text method | NSD-trained vision decoders · NSD-Imagery mental imagery trials | human identification accuracy: 52.89% | No clear effect |
| Evaluate the same stimuli with mental imagery rather than vision trials | NSD-trained reconstruction methods · NSD-Imagery | reconstruction performance | Hurt |
| Use a more complex vision-decoding architecture | NSD-trained vision decoders · NSD-Imagery mental imagery trials | generalization performance | Hurt |
| Improve seen-image decoding performance | NSD-trained vision decoders · NSD-Imagery mental imagery trials | reconstruction quality: MindEye1: r = 0.16; Brain Diffuser: r = 0.19; iCNN: r = 0.22; MindEye2: r = 0.13 | Mixed |
Surprising findings
- Vision-decoding performance (MindEye2, SOTA on seen images) does not predict mental imagery decoding; MindEye2 generalizes extremely poorly to imagery, near chance by human raters.
- Mental imagery reconstructions of complex natural scenes were rated similarly to or sometimes better than reconstructions of seen simple stimuli, suggesting training-distribution alignment matters more than stimulus complexity.
Limitations
- Mental imagery dataset is small (18 stimuli), precluding large-scale training on imagery data and requiring cross-decoding.
- Takagi et al. method performed at chance on both vision and imagery trials, so it was excluded from further analyses.
- Only subjects 1, 2, 5, and 7 completed the full NSD experiment, limiting the training data for the evaluated methods.
Experimental setup
Datasets: NSD NSD-Imagery COCO
Benchmarks: NSD-Imagery
Metrics: PixCorr SSIM AlexNet 2-way comparison Inception 2-way comparison CLIP 2-way comparison EffNet-B distance SwAV distance brain correlation human two-alternative identification accuracy human similarity ratings
Baselines: MindEye1 MindEye2 Brain Diffuser iCNN Takagi et al.
Builds on: Natural Scenes Dataset MindEye Brain Diffuser iCNN
Closest papers by meaning
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.92)
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.89)
- $i$MIND: Insightful Multi-subject Invariant Neural Decoding (NeurIPS 2025 · similarity 0.89)
- MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data (ICML 2024 · similarity 0.89)
- MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction (AAAI 2025 · similarity 0.88)
- MindBridge: A Cross-Subject Brain Decoding Framework (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.