AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

UMBRAE: Unified Multimodal Brain Decoding

Weihao Xia, Raoul de Charette, A. Cengiz Oztireli, Jing-Hao Xue

ECCV 2024 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 31 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add cross-subject training (universal encoder over S1, S2, S5, S7) (ablation)UMBRAE · BrainHub captioning (S1)CIDEr: 51.93 -> 61.06Improved
add cross-subject training (ablation)UMBRAE · BrainHub grounding (S1, all)acc@0.5: 13.72 -> 18.93Improved
use transformer brain encoder instead of MLP encoder (ablation)UMBRAE · BrainHub captioning (S1)CIDEr: 46.24 -> 51.93Improved
use transformer encoder instead of MLP encoder (parameters) (ablation)UMBRAE · model sizeparameters (M): 1,003.64 -> 112.63 (single subject)Improved
finetune the adapter or train it jointly with the encoder (ablation)UMBRAE · BrainHub captioning (S1)CIDEr: 51.93 -> 43.53Hurt
align brain with LLM embedding instead of image features (ablation)UMBRAE · BrainHub captioning (S1)CIDEr: 51.93 -> 36.40Hurt
use contrastive (NCE) loss instead of MSE reconstruction loss (ablation)UMBRAE · BrainHub captioning (S1)CIDEr: 51.93 -> 3.16Hurt
adapt to new subject S7 with 30% of data (finetuned) (ablation)UMBRAE · subject adaptation (S7)captioning metricsNo clear effect
freeze the universal encoder and train only the tokenizer for new subject (ablation)UMBRAE · subject adaptation (S7)captioning metricsHurt
use UMBRAE versus constructed Shikra-based baselines for groundingUMBRAE · BrainHub grounding (S1, all)inference time (s): 16.4 -> 0.92Improved

Later papers that compared with this method

9 later accepted papers report UMBRAE as a baseline in their results tables; 7 of them report a better number on most of the results they share with it.

1 more not listed.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy