Research map / Vision-language and multimodal / Human and embodied modeling
MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MindAligner, which learns a low-rank Brain Transfer Matrix mapping a new subject's fMRI to a seen subject's space (with a cross-stimulus mapper and multi-level alignment losses) so a pre-trained MindEye2-style decoder can be reused from one hour of data.
Problem
Cross-subject fMRI visual decoding fails to generalize to new subjects under limited data (one NSD session), and implicit latent alignment across subjects is suboptimal and uninterpretable.
Approach
Explicitly map a novel subject's voxel-level fMRI to a known subject's voxel space via a learned low-rank Brain Transfer Matrix, trained without shared stimuli using a cross-stimulus neural mapper (FiLM-style modulation conditioned on CLIP image differences) and multi-level alignment losses (reconstruction, KL, latent similarity), so the pre-trained decoder is reused unchanged.
Key results
- With 1 hour of data, MindAligner reaches brain retrieval 75.3% vs 57.4% for MindEye2 (1 h), and Alex(2) 85.6% vs 84.2%.
- Trainable parameters are about 6% of MindEye2's (139.23M vs 2.21G), with inference time 5.056 s vs 5.000 s per image.
- Explicit alignment yields a reported 17.9% improvement in brain retrieval over the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace implicit shared-space alignment with explicit subject-to-subject brain alignment | MindAligner vs MindEye2 · NSD, 1 h | Brain retrieval: 57.4% -> 75.3% | Helped |
| Replace implicit shared-space alignment with explicit subject-to-subject brain alignment | MindAligner vs MindEye2 · NSD, 1 h | PixCorr: 0.195 -> 0.206 | Helped |
| Add signal reconstruction loss L_rec (ablation) | MindAligner · NSD, Subject 2 novel / Subject 1 known | PixCorr: 0.072 -> 0.186 | Helped |
| Add KL distribution-alignment loss L_KL (ablation) | MindAligner with L_dec and L_rec · NSD, Subject 2 novel / Subject 1 known | PixCorr: 0.186 -> 0.191 | Helped |
| Add latent alignment loss L_latent (ablation) | MindAligner with L_dec, L_rec, and L_KL · NSD, Subject 2 novel / Subject 1 known | CLIP: 82.16% -> 82.72% | Helped |
| Add latent alignment loss L_latent (ablation) | MindAligner with L_dec, L_rec, and L_KL · NSD, Subject 2 novel / Subject 1 known | Incep: 86.61% -> 86.24% | Hurt |
| Increase BTM hidden size from 64 to 4096 (ablation) | MindAligner · NSD, Subject 2 novel / Subject 1 known | PixCorr: 0.144 -> 0.218 | Helped |
| Use hidden size 1024 instead of 4096 (ablation) | MindAligner · NSD, Subject 2 novel / Subject 1 known | PixCorr: 0.218 -> 0.204 | Hurt |
Surprising findings
- Reconstruction results are stable regardless of which known subject the new subject is aligned to.
Limitations
- Evaluated only on NSD subjects 1, 2, 5, 7 with a single-session budget.
- Relies on a pre-trained MindEye2 decoder for known subjects; alignment is not evaluated beyond this setup.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) MSCOCO-2017
Benchmarks: NSD visual decoding
Metrics: PixCorr SSIM AlexNet(2) AlexNet(5) Inception CLIP EffNet-B SwAV retrieval accuracy fMRI spatial correlation
Baselines: MindEye2 MindBridge
Builds on: MindEye2 MindBridge FiLM
Closest papers by meaning
- MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction (AAAI 2025 · similarity 0.92)
- MindBridge: A Cross-Subject Brain Decoding Framework (CVPR 2024 · similarity 0.91)
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.91)
- Wills Aligner: Multi-Subject Collaborative Brain Visual Decoding (AAAI 2025 · similarity 0.90)
- MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data (ICML 2024 · similarity 0.90)
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.