Research map / Vision-language and multimodal / Human and embodied modeling
$i$MIND: Insightful Multi-subject Invariant Neural Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes iMIND, a multi-subject fMRI decoder that pretrains a ViT masked autoencoder, learns an orthonormal basis splitting neural tokens into subject-specific and object-specific subspaces, and performs biometric and multi-label semantic decoding on NSD.
Problem
Existing fMRI visual decoding work emphasizes image reconstruction that relies on pretrained generative priors and gives little insight into voxel-level object encoding; multi-subject models also entangle subject-specific and object-related signals.
Approach
Re-represent latent neural tokens in a learnable orthonormal basis that separates subject-specific and object-specific subspaces, classify subjects from one part and objects (multi-label) from the other, using frozen CLIP vision features as cross-attention queries over the object subspace.
Key results
- Semantic decoding on NSD: fMRI-only iMIND reaches mAP .310 and AUC .913, versus .258 mAP / .877 AUC for the best prior multi-subject baseline (CLIP-MUSED).
- Biometric (subject) decoding: iMIND reaches ACC .999 and MCC .999, against at most .283 ACC for the best baseline (MLP with cross-entropy).
- With fMRI plus image input, iMIND reports mAP .784 and AUC .984; the paper notes the ablation mAP changes are reported as percentage-point drops.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the orthonormal loss L_orth (ablation) | iMIND · NSD semantic decoding | mAP: -7.15 (↓) | Helped |
| Add subject classification loss L_subj (ablation) | iMIND · NSD semantic decoding | mAP: -11.17 (↓) | Helped |
| Add both L_subj and L_orth (ablation) | iMIND · NSD semantic decoding | mAP: -11.07 (↓) | Helped |
| Set lambda to .01 instead of .1 (ablation) | iMIND · NSD semantic decoding | mAP: -0.94 (↓) | Hurt |
| Set lambda to 1 instead of .1 (ablation) | iMIND · NSD semantic decoding | mAP: -2.95 (↓) | Hurt |
| Set lambda to 10 instead of .1 (ablation) | iMIND · NSD semantic decoding | mAP: -0.84 (↓) | Hurt |
| Vary attention-head count from 4 to 1, 2, 6, or 8 (ablation) | iMIND · NSD semantic decoding | mAP: -1.95; -1.12; -1.35; -9.18 | Hurt |
| Reduce d_obj from 700 to 100, 200, 300, 400, 500, or 600 (ablation) | iMIND · NSD semantic decoding | mAP: -4.01; -2.11; -1.04; -1.38; -0.89; -1.14 | Hurt |
| Use fMRI+Image rather than fMRI-only decoding | iMIND · NSD semantic decoding | mAP: .310 -> .784 | Helped |
Surprising findings
- Subject classification from fMRI is near-perfect (ACC .999), which the authors interpret as biometric information being highly discriminative across subjects.
- The reported fMRI+Image result (mAP .784) is far above the fMRI-only result (.310), a large gap attributed to the CLIP-query cross-attention design.
Limitations
- Pretrained MAE reconstructions underperform ground-truth voxel signals, suggesting the MAE backbone may not be optimal.
- Flattening voxels discards spatial relationships among neighboring voxels.
- Interpretability relies on post hoc Grad-CAM and Rollout methods, which give only approximate explanations.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) MS-COCO
Benchmarks: NSD
Metrics: mAP AUC Hamming distance ACC MCC
Baselines: MLP ViT EMB SMODEL-MLP SMODEL-ViT CLIP-MUSED K-Means Linear regression classifier
Builds on: SC-MBM MinD-Vis CLIP
Closest papers by meaning
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.90)
- MindBridge: A Cross-Subject Brain Decoding Framework (CVPR 2024 · similarity 0.90)
- Toward Generalizing Visual Brain Decoding to Unseen Subjects (ICLR 2025 · similarity 0.90)
- CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding (ICLR 2024 · similarity 0.89)
- NSD-Imagery: A Benchmark Dataset for Extending fMRI Vision Decoding Methods to Mental Imagery (CVPR 2025 · similarity 0.89)
- Multigranular Evaluation for Brain Visual Decoding (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.