Research map / Vision-language and multimodal / Cross-modal learning
Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Embracing Aleatoric Uncertainty (EAU), a multimodal fusion method that models unimodal aleatoric uncertainty with Gaussian embeddings, uses it in contrastive learning for stable unimodal features, and applies a VIB-based compression for robust joint representations.
Problem
Multimodal fusion degrades when unimodal data contain noise (aleatoric uncertainty), which corrupts cross-modal interactions and hurts robustness.
Approach
Represent each unimodal sample as a multivariate Gaussian whose variance captures aleatoric uncertainty, use the variance for uncertainty-based contrastive augmentation (SUFA), then fuse with uncertainty-weighted attention and compress the joint representation with a variational information bottleneck (RMFI).
Key results
- On noisy MVSA-Single (Gaussian noise eps=10), EAU reaches 62.04% vs. 61.28% for QMF.
- On NYU Depth v2 clean, EAU achieves 72.05% accuracy vs. 70.09% for QMF.
- On CMU-MOSI, EAU reports Acc7 48.8, F1 86.2, Corr 0.809, the best among listed methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply EAU fusion | multimodal sentiment analysis · CMU-MOSI | Acc7: 48.6 -> 48.8 | Helped |
| Apply EAU fusion | multimodal sentiment analysis · CMU-MOSEI | Acc7: 54.9 -> 54.8 | No clear effect |
| Apply EAU fusion | multimodal image classification · MVSA-Single | Acc: 78.50 -> 79.15 | Helped |
| Apply EAU fusion | multimodal image classification · Food-101 | Acc: 92.92 -> 93.20 | Helped |
| Apply EAU fusion | multimodal image classification · NYU Depth v2 | Acc: 71.06 -> 72.05 | Helped |
| Apply EAU fusion | noisy multimodal input · Noisy NYU Depth v2, Gaussian ϵ = 10 | Acc: 55.60 -> 58.85 | Helped |
| Add SUFA to RMFI (ablation) | EAU on noisy MVSA-Single · Noisy MVSA-Single, Salt-Pepper ϵ = 10 | Acc: 59.11 ± 1.92 -> 61.04 ± 1.31 | Helped |
| Replace Late Fusion with RMFI (ablation) | SUFA on noisy MVSA-Single · Noisy MVSA-Single, Salt-Pepper ϵ = 10 | Acc: 55.93 ± 2.25 -> 61.04 ± 1.31 | Helped |
| Replace Concat with RMFI (ablation) | SUFA on noisy MVSA-Single · Noisy MVSA-Single, Salt-Pepper ϵ = 10 | Acc: 60.67 ± 1.38 -> 61.04 ± 1.31 | Helped |
| Add audio modality to text (ablation) | EAU on CMU-MOSI · CMU-MOSI | Acc7: 44.1 -> 46.7 | Helped |
| Add video modality to text (ablation) | EAU on CMU-MOSI · CMU-MOSI | Acc: 76.30 -> 79.15 | Helped |
Surprising findings
- Improvements on UPMC Food101 are smaller than on other datasets, which the authors attribute to clean image-text pairs limiting augmentation benefit.
Limitations
- Gains on UPMC Food101 are modest because clean inputs limit the augmentation benefit.
- Improvement on NYU Depth v2 is smaller than on MVSA-Single, as depth offers less complementary information than text.
Experimental setup
Datasets: CMU-MOSI CMU-MOSEI MVSA-Single UPMC Food101 NYU Depth v2
Benchmarks: CMU-MOSI CMU-MOSEI MVSA-Single UPMC Food101 NYU Depth v2
Metrics: Acc7 F1 Pearson correlation accuracy
Baselines: MIB HMA MIM GCNet ConFEDE DiCMoR DMD MMBT TMC ITIN PMF MVCN
Builds on: QMF Variational Information Bottleneck dynamic multimodal fusion MMBT
Closest papers by meaning
- Multimodal Representation Learning by Alternating Unimodal Adaptation (CVPR 2024 · similarity 0.88)
- Beyond Modality Fusion: Deep Ensembles for Multimodal Classification (NeurIPS 2026 · similarity 0.87)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.86)
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.86)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.86)
- Data-Efficient Multimodal Fusion on a Single GPU (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.