Research map / Vision-language and multimodal / Human and embodied modeling
Wills Aligner: Multi-Subject Collaborative Brain Visual Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Wills Aligner, a multi-subject fMRI visual decoding framework that combines anatomical alignment to fsaverage7, subject-guided sparse MoBE LoRA-style adapters with a meta-learning schedule, and a semantic relation alignment loss, evaluated on NSD for classification, retrieval, few-shot, and reconstruction.
Problem
Single-subject fMRI visual decoding models do not generalize across subjects because of cortical structural differences and individual fMRI pattern differences; existing multi-subject methods underperform single-subject ones and ignore inter-subject commonality.
Approach
Align fMRI to a standard cortical template (anatomical alignment), route each subject's data through subject-specific low-rank MoBE adapters via a global subject-classifier router, use a semantic relation alignment loss to learn subject-invariant commonality, and alternate adapter and backbone updates with meta-learning.
Key results
- Classification on NSD: mAP 0.424 and AUC 0.937 with 19M parameters, versus 0.258 mAP and 0.877 AUC for CLIP-MUSED.
- Retrieval on Subj01: image retrieval 95.4% versus 83.7% for MindEye (single-subject, contrastive loss only), using 523M parameters.
- Few-shot (1 session) classification on Subj01 raises mAP from 0.128 (vanilla) to 0.270.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MoBE adapters and corresponding meta-learning (ablation) | Wills Aligner · NSD multi-label classification | mAP: 0.361 -> 0.424 | Helped |
| Add SRA loss (ablation) | Wills Aligner · NSD multi-label classification | mAP: 0.417 -> 0.424 | Helped |
| Add MoBE adapters and corresponding meta-learning (ablation) | Wills Aligner · NSD image reconstruction | PixCorr: 0.274 -> 0.271 | Hurt |
| Add SRA loss (ablation) | Wills Aligner · NSD image reconstruction | PixCorr: 0.274 -> 0.271 | Hurt |
| Replace anatomical alignment with random voxel-dropout misalignment (ablation) | Wills Aligner · NSD classification | mAP: 0.424 -> 0.247 | Hurt |
| Vary MoBE adapter rank from 2 to 32 (ablation) | Wills Aligner · NSD classification | mAP: 0.461; 0.452; 0.445; 0.467; 0.448 | No clear effect |
| Use Wills Aligner instead of MindEye with Contrastive Loss + MSE loss | NSD retrieval, Subj01 · NSD cross-modal retrieval, Subj01 | Image retrieval accuracy: 88.8% -> 95.4% | Helped |
| Use Wills Aligner instead of MindEye | NSD image reconstruction · NSD fMRI-to-image reconstruction | PixCorr: 0.309 -> 0.271 | Hurt |
| Use Wills Aligner instead of Mind-Diffuser | NSD image reconstruction · NSD fMRI-to-image reconstruction | SSIM: 0.356 -> 0.328 | Hurt |
| Use Wills Aligner instead of DREAM | NSD image reconstruction · NSD fMRI-to-image reconstruction | CLIP: 95.2% -> 94.8% | Hurt |
Surprising findings
- Multi-subject vanilla training performs much worse than the single-subject vanilla model on classification (mAP 0.150 vs 0.258).
- Reconstruction gains do not extend to low-level pixel metrics; PixCorr and SSIM are not better than single-subject baselines.
Limitations
- Reconstruction does not improve low-level pixel metrics such as PixCorr and SSIM.
- Retrieval uses a 523M-parameter model, so parameter count is not uniformly lower than baselines.
- Router-based sparse routing is effectively dense with high classifier confidence rather than truly sparse.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) MSCOCO LAION-5B
Benchmarks: NSD LAION-5B
Metrics: mAP AUC Hamming distance top-1 retrieval accuracy PixCorr SSIM AlexNet two-way identification Inception CLIP EfficientNet-B1 distance
Baselines: Single-Subject Vanilla Method Multi-Subject Vanilla Method EMB CLIP-MUSED Mind Reader Brain Diffuser MindEye Mind-Diffuser DREAM MindBridge Lite-Mind
Builds on: LoRA MindEye CLIP-MUSED MindBridge
Closest papers by meaning
- MindAligner: Explicit Brain Functional Alignment for Cross-Subject Visual Decoding from Limited fMRI Data (ICML 2025 · similarity 0.90)
- MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction (AAAI 2025 · similarity 0.89)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.88)
- MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding (NeurIPS 2026 · similarity 0.88)
- Human-Aligned Image Models Improve Visual Decoding from the Brain (ICML 2025 · similarity 0.88)
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.