Research map / Vision-language and multimodal / Cross-modal learning

MIRAGE: Adaptive Multimodal Gating for Whole-Brain fMRI Encoding

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MIRAGE, a whole-brain fMRI encoder that uses natively multimodal Qwen3-Omni features with learned cross-attention layer pooling and a transformer brain encoder, showing native fusion beats post-hoc fusion and reaching SOTA on Algonauts 2025.

Problem

Predicting whole-brain fMRI responses to naturalistic audiovisual movies from unimodal or late-fused features, with rigid layer selection that ignores depth differences across regions.

Approach

Use features from a single omni-modal foundation model (native fusion inside the language module) and learn per-modality cross-attention layer poolers with latent queries over all backbone layers, followed by a shared transformer encoder and subject-specific parcel readouts.

Key results

• MIRAGE ensemble reaches r=0.323 on Friends S07 and r=0.227 on OOD movies, above published Algonauts 2025 ensembles (max 0.215 OOD).
• Native fusion beats post-hoc fusion at every level; linear ridge OOD r rises from 0.090 to 0.134 with Qwen3-Omni native features.
• Single MIRAGE model achieves r=0.310 on S07 and r=0.217 on OOD, versus 0.196 for TRIBE v1 on OOD.

Surprising findings

• Early Qwen3-Omni layers receive near-zero attention weight; vision attends sharply to layers 25-30.

Limitations

• Only four subjects, limiting population generalization and cross-subject statistical power.
• Native-fusion result is established only within the Qwen-Omni family and 3B active parameters.
• Caching all layers raises feature storage and dataloader cost; attention attributions are model-level not parcel-level.

Experimental setup

Datasets: Algonauts 2025 challenge data Courtois NeuroMod (Friends, Movie10)

Benchmarks: Algonauts 2025 (Friends S06 validation, Friends S07 test, OOD movies)

Metrics: Pearson correlation per parcel averaged over parcels and sub

Baselines: Linear ridge (post-hoc fusion) Linear (native fusion) TRIBE v1 TRIBE v2 MedARC ModalityRNN VIBE

Builds on: TRIBE Perceiver Algonauts 2025 challenge baselines

Closest papers by meaning

• Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG (NeurIPS 2026 · similarity 0.88)
• MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding (NeurIPS 2025 · similarity 0.88)
• BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction (NeurIPS 2026 · similarity 0.88)
• Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction (NeurIPS 2024 · similarity 0.87)
• Revealing Vision-Language Integration in the Brain with Multimodal Networks (ICML 2024 · similarity 0.87)
• Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex (NeurIPS 2025 · similarity 0.87)