Research map / Vision-language and multimodal / Cross-modal learning

Beyond Modality Fusion: Deep Ensembles for Multimodal Classification

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that deep ensembles of unimodal networks (heterogeneous or homogeneous, with no explicit fusion) beat imbalance-aware late-fusion, intermediate-fusion, and hybrid methods at equal parameter count, and proposes a logit-ratio heuristic for allocating ensemble members across modalities.

Problem

Late-fusion multimodal classifiers often underperform the best unimodal model under modality imbalance, and existing imbalance-aware fixes are not compared fairly against simple alternatives at equal parameter count.

Approach

Train independent unimodal networks on each modality and average their logits (heterogeneous deep ensemble), with the number of networks per modality set by the inverse validation-loss ratio, avoiding any explicit modality fusion.

Key results

• On synthetic bimodal CIFAR-100 (0.05-0.95), HoDE(2) reaches 78.5% vs 34.0% for vanilla late fusion; at (0.5-0.5), HeDE(2) reaches 86.9% vs 82.5% for late fusion.
• On CREMA-D (clean split), HeDE(2) gets 71.2% vs 56.7% for late fusion and 62.8% for HoDE(2).
• Deep ensembles outperform I2M2 on all five settings; e.g. ImageNet-1K (0.01-0.99) HeDE(4) 77.7% vs I2M2 62.4%.

Surprising findings

• Under extreme modality imbalance, a late-fusion encoder trained on the weaker modality can match or exceed a unimodal network trained on it, suggesting the late-fusion network underexplores the stronger modality.
• Many existing late-fusion methods reported on CREMA-D suffer from actor-level data leakage; accuracies drop by 6.7 to 16.9 points on a clean split.

Limitations

• Synthetic dataset is class-aligned rather than instance-aligned, with no instance-level complementary information.
• Convergence speed and predictive strength of a modality can only be varied together in the synthetic construction.
• Heuristic lacks theoretical justification; evaluation on trimodal real-world data is missing.

Experimental setup

Datasets: CIFAR-100 ImageNet-1K CREMA-D Sarcasm Kinetics-400

Benchmarks: CIFAR-100 (synthetic bimodal) ImageNet-1K (synthetic bimodal/trimodal) CREMA-D Sarcasm Kinetics-400

Metrics: accuracy log-loss expected calibration error

Baselines: InfoReg MMPareto OGM-GE AUG ReconBoost MMTM I2M2 LFM

Builds on: deep ensembles MMTM I2M2 OGM-GE MMPareto InfoReg

Closest papers by meaning

• Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.90)
• Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion (NeurIPS 2025 · similarity 0.88)
• Facilitating Multimodal Classification via Dynamically Learning Modality Gap (NeurIPS 2024 · similarity 0.88)
• Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Process (NeurIPS 2025 · similarity 0.88)
• CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning (NeurIPS 2025 · similarity 0.88)
• MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning (NeurIPS 2025 · similarity 0.88)