Research map / Vision-language and multimodal / Cross-modal learning

Fusion or Confusion? Multimodal Complexity Is Not All You Need

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Reimplements 19 multimodal methods plus a simple late-fusion transformer baseline (SimBaMM) across nine datasets under standardized protocols and finds complex multimodal architectures do not reliably beat it or unimodal baselines.

Problem

Whether architectural complexity in multimodal learning yields real performance gains over simple late-fusion baselines and unimodal models when evaluation is controlled (tuning, initialization, subject-wise CV, statistical testing).

Approach

Proposes SimBaMM, a simple late-fusion transformer over per-modality encoder tokens with missing-modality attention masking, and uses it as the reference in a large, unified re-evaluation of 19 recent multimodal methods with Bayesian hierarchical comparison and ROPE-based decision analysis.

Key results

• No complex method reached a high probability (max about 0.30, C-Mamba) of outperforming SimBaMM-CLS; for most methods practical equivalence or SimBaMM-CLS superiority was found.
• SimBaMM-CLS (~0.53) matched the top complex methods in probability of beating the best unimodal baseline, while DGL, OMIB, LMF and PDF were likely worse than the best unimodal model (>80%).
• In the CREMA-D case study, correcting for subject leakage and tuning hyperparameters reversed the ranking so SimBaMM matched or exceeded AUG on multimodal performance.

Surprising findings

• Many complex methods, including recent ones, are likely worse than simply using the best single modality.
• MulT and LMF become unstable or infeasible with many modalities (MulT OOM on UK Biobank; LMF near-random AUROC there).

Limitations

• Excludes early fusion, pre-training, contrastive, and vision-language methods, so the conclusions are scoped to the settings studied.
• Standardized tuning starting from SimBaMM settings may introduce tuning bias toward SimBaMM.
• Conclusions depend on chosen ROPE width and preprocessing; no parameter-matched analysis was performed.

Experimental setup

Datasets: MIMIC Symile MIMIC HAIM INSPECT UK Biobank CMU-MOSI CMU-MOSEI CH-SIMS CH-SIMS 2 CREMA-D

Benchmarks: MIMIC HAIM MIMIC Symile INSPECT UK Biobank CMU-MOSI CMU-MOSEI CH-SIMS CH-SIMS 2 CREMA-D

Metrics: AUROC accuracy (Acc7/Acc5/Acc6) P(base>other)/P(ROPE)/P(other>base) Bayesian posterior FLOPs parameters training time

Baselines: IMDer MMP ShaSpec RegBN AUG MBT LMF PDF C-Mamba OMIB MulT OGM

Builds on: ScheduleFree AdamW OGM-GE AUG IMDer MultiBench

Closest papers by meaning

• Beyond Modality Fusion: Deep Ensembles for Multimodal Classification (NeurIPS 2026 · similarity 0.90)
• Beyond task performance: evaluating and reducing the flaws of large multimodal models with in-context-learning (ICLR 2024 · similarity 0.88)
• Scaling Laws for Multimodal Data Mixtures (NeurIPS 2026 · similarity 0.88)
• Revisiting the Data Sampling in Multimodal Post-training from a Difficulty-Distinguish View (AAAI 2026 · similarity 0.88)
• Balancing Multimodal Training Through Game-Theoretic Regularization (NeurIPS 2025 · similarity 0.88)
• Improving Multimodal Learning Balance and Sufficiency through Data Remixing (ICML 2025 · similarity 0.88)