Research map / Vision-language and multimodal / Cross-modal learning
UMBRAE: Unified Multimodal Brain Decoding
Weihao Xia, Raoul de Charette, A. Cengiz Oztireli, Jing-Hao Xue
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-subject training (universal encoder over S1, S2, S5, S7) (ablation) | UMBRAE · BrainHub captioning (S1) | CIDEr: 51.93 -> 61.06 | Improved |
| add cross-subject training (ablation) | UMBRAE · BrainHub grounding (S1, all) | acc@0.5: 13.72 -> 18.93 | Improved |
| use transformer brain encoder instead of MLP encoder (ablation) | UMBRAE · BrainHub captioning (S1) | CIDEr: 46.24 -> 51.93 | Improved |
| use transformer encoder instead of MLP encoder (parameters) (ablation) | UMBRAE · model size | parameters (M): 1,003.64 -> 112.63 (single subject) | Improved |
| finetune the adapter or train it jointly with the encoder (ablation) | UMBRAE · BrainHub captioning (S1) | CIDEr: 51.93 -> 43.53 | Hurt |
| align brain with LLM embedding instead of image features (ablation) | UMBRAE · BrainHub captioning (S1) | CIDEr: 51.93 -> 36.40 | Hurt |
| use contrastive (NCE) loss instead of MSE reconstruction loss (ablation) | UMBRAE · BrainHub captioning (S1) | CIDEr: 51.93 -> 3.16 | Hurt |
| adapt to new subject S7 with 30% of data (finetuned) (ablation) | UMBRAE · subject adaptation (S7) | captioning metrics | No clear effect |
| freeze the universal encoder and train only the tokenizer for new subject (ablation) | UMBRAE · subject adaptation (S7) | captioning metrics | Hurt |
| use UMBRAE versus constructed Shikra-based baselines for grounding | UMBRAE · BrainHub grounding (S1, all) | inference time (s): 16.4 -> 0.92 | Improved |
Later papers that compared with this method
9 later accepted papers report UMBRAE as a baseline in their results tables; 7 of them report a better number on most of the results they share with it.
- BrainJanus: A Unified Model for Understanding and Generation across Brain, Vision, and Language (ICML 2026 · better on 7 of 7 shared results)
- Compositional Neuro-Symbolic Concepts in Neural Activities (ICLR 2026 · better on 5 of 5 shared results)
- Reducing Semantic Mismatch in Brain-to-Text Decoding Through Personalized Multimodal Masking (ICLR 2026 · better on 2 of 3 shared results)
- Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion (ICML 2026 · better on 0 of 1 shared result)
- Beyond Brain Decoding: Visual-Semantic Reconstructions to Mental Creation Extension Based on fMRI (ICCV 2025 · better on 4 of 4 shared results)
- Exploring The Visual Feature Space for Multimodal Neural Decoding (ICCV 2025 · better on 2 of 3 shared results)
- MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-text Decoding (ICML 2025 · better on 2 of 2 shared results)
- Bridging the Gap between Brain and Machine in Interpreting Visual Semantics: Towards Self-adaptive Brain-to-Text Decoding (ICCV 2025 · better on 2 of 2 shared results)
1 more not listed.
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- CSBrain: A Cross-scale Spatiotemporal Brain Foundation Model for EEG Decoding (NeurIPS 2025 · similarity 0.88)
- MindBridge: A Cross-Subject Brain Decoding Framework (CVPR 2024 · similarity 0.87)
- Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.87)
- $i$MIND: Insightful Multi-subject Invariant Neural Decoding (NeurIPS 2025 · similarity 0.87)
- MindAlign: Bridging EEG, Vision, and Language for Zero-Shot Visual Decoding (NeurIPS 2026 · similarity 0.87)
- Multigranular Evaluation for Brain Visual Decoding (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
