Research map / Vision-language and multimodal / Human and embodied modeling
MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoRE-Brain, a hierarchical mixture-of-experts fMRI encoder routing voxel groups to CLIP embeddings, plus time/space routers that condition a LoRA-tuned SDXL for interpretable, cross-subject fMRI-to-image reconstruction on NSD.
Problem
Existing fMRI visual decoders treat fMRI as monolithic input, need subject-specific training for cross-subject generalization, and offer little mechanistic interpretability of how brain regions shape reconstructions.
Approach
A hierarchical MoE encoder with data-driven voxel-to-expert routing (top-k per expert, 4 levels, 16 final experts) that maps fMRI to CLIP space; a Time Router picks expert levels across diffusion timesteps (with coarse-to-fine prior) and a Space Router cross-attends over selected experts; only routers are finetuned for new subjects while experts are frozen.
Key results
- Bottleneck analysis: MoRE-Brain shows the largest degradation in SSIM, DreamSim and InceptionV3 as fMRI bottleneck shrinks, versus MindEye2 and MindBridge, suggesting heavier reliance on fMRI signal.
- Ablation: time+space routers (TS) with image conditioning reach DreamSim 0.507 and InceptionV3 0.962 and SSIM 0.415 (combined text/image conditioning), the best across the ablated variants.
- Parameter efficiency: finetuning only routers trains 44.84% of MoRE-Brain's 293.4M parameters versus 100% of MindEye2's 729.3M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add image conditioning instead of text conditioning only (ablation) | MoRE-Brain · NSD | SSIM: 0.410 -> 0.422 | Helped |
| add time and space routers to text conditioning (ablation) | MoRE-Brain · NSD | Incep: 0.926 -> 0.948 | Helped |
| add time and space routers to image conditioning (ablation) | MoRE-Brain · NSD | SSIM: 0.422 -> 0.382 | Hurt |
| add both text and image conditioning (ablation) | MoRE-Brain · NSD | SSIM: 0.410 -> 0.407 | No clear effect |
| add the time router while keeping the space router (ablation) | MoRE-Brain · NSD | SSIM: 0.375 -> 0.407 | Helped |
| use a fixed time-routing schedule instead of no router (ablation) | MoRE-Brain · NSD | DreamSim: 0.585 -> 0.533 | Helped |
| use learned soft time selection rather than fixed scheduling (ablation) | MoRE-Brain · NSD | DreamSim: 0.533 -> 0.543 | Helped |
| use learned hard time selection instead of learned soft selection (ablation) | MoRE-Brain · NSD | Alex (2): 0.764 -> 0.650 | Hurt |
| project image embeddings to text space for text conditioning (ablation) | MoRE-Brain · NSD | SSIM: 0.415 -> 0.384 | Hurt |
Surprising findings
- MindEye2 keeps high CLIP-Cos (~0.8 even for random fMRI input) across all bottleneck sizes, suggesting its outputs are dominated by generative/semantic priors rather than fMRI.
- MindBridge shows relatively robust pixel-space metrics under narrow bottlenecks, indicating weaker dependence on fMRI.
Limitations
- Evaluated only on NSD with a single training subject for the full model; cross-subject results rely on router finetuning.
- Interpretability relies on GradientSHAP attributions and ICA, which are post-hoc and not validated against independent neuroscience ground truth.
- Some metrics favor baselines (e.g., MindBridge on SSIM/InceptionV3 robustness; MindEye2 CLIP-Cos), and the refinement step is optional.
Experimental setup
Datasets: Natural Scenes Dataset (NSD) LAION-2B (CLIP pretraining) COCO captions/supercategories
Benchmarks: NSD shared1000 test set
Metrics: SSIM AlexNet(2) InceptionV3 DreamSim CLIP cosine similarity bottleneck analysis parameter count
Baselines: MindEye2 MindBridge
Builds on: Stable Diffusion XL CLIP MindEye2 MindBridge IP-Adapter
Closest papers by meaning
- Toward Generalizing Visual Brain Decoding to Unseen Subjects (ICLR 2025 · similarity 0.91)
- See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI (AAAI 2025 · similarity 0.90)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.90)
- MindBridge: A Cross-Subject Brain Decoding Framework (CVPR 2024 · similarity 0.89)
- A Convolutional Neural Network Interpretable Framework for Human Ventral Visual Pathway Representation (AAAI 2024 · similarity 0.89)
- Brain decoding: toward real-time reconstruction of visual perception (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.