Research map / Vision-language and multimodal / Cross-modal learning
What Makes Multi-Modal Learning Better than Single (Provably)
Yu Huang, Chenzhuang Du, Zihui Xue, Xuanyao Chen, Hang Zhao, Longbo Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Video to Text | IEMOCAP late-fusion classifier · IEMOCAP test | top-1 accuracy: 49.93$\pm$0.57 -> 51.08$\pm$0.66 | Improved |
| Add Audio to Text | IEMOCAP late-fusion classifier · IEMOCAP test | top-1 accuracy: 49.93$\pm$0.57 -> 53.03$\pm$0.21 | Improved |
| Add Audio to Text + Video | IEMOCAP late-fusion classifier · IEMOCAP test | top-1 accuracy: 51.08$\pm$0.66 -> 53.89 $\pm$ 0.47 | Improved |
| Add modalities beyond m1 | Synthetic-data MLP · Synthetic data, overlap 0.0 | MSE loss: 301.92$\pm$7.85 -> 207.45$\pm$4.68 -> 103.23$\pm$0.61 -> 0 | Improved |
| Add modalities beyond m1 | Synthetic-data MLP · Synthetic data, overlap 0.2 | MSE loss: 193.28$\pm$1.08 -> 129.81$\pm$4.36 -> 65.17$\pm$1.52 -> 0 | Improved |
| Add modalities beyond m1 | Synthetic-data MLP · Synthetic data, overlap 0.5 | MSE loss: 75.89$\pm$1.28 -> 51.25$\pm$1.06 -> 26.06$\pm$0.69 -> 0 | Improved |
| Add modalities beyond m1 | Synthetic-data MLP · Synthetic data, overlap 0.8 | MSE loss: 12.04$\pm$0.39 -> 8.16$\pm$0.17 -> 4.18$\pm$0.05 -> 0 | Improved |
| Add modalities beyond m1 | Synthetic-data MLP · Synthetic data, overlap 1 | MSE loss: 0 -> 0 -> 0 -> 0 | No clear effect |
| Use all three modalities rather than Text alone | IEMOCAP late-fusion classifier · IEMOCAP test, sample ratio $10^{-3}$ | top-1 accuracy: 29.08$\pm$3.34 -> 32.24$\pm$1.17 | Mixed |
| Use all three modalities rather than Text alone | IEMOCAP late-fusion classifier · IEMOCAP test, sample ratio $10^{-2}$ | top-1 accuracy: 45.63$\pm$0.29 -> 46.39$\pm$3.82 | Mixed |
| Increase the training sample ratio | IEMOCAP modality settings · IEMOCAP test | top-1 accuracy: T: 23.66$\pm$1.28 -> 49.93$\pm$0.57; TA: 25.06$\pm$1.05 -> 51.08$\pm$0.66; TV: 24.71$\pm$0.87 -> 53.03$\pm$0.21; TVA: 24 | Improved |
Closest papers by meaning
- On the Comparison between Multi-modal and Single-modal Contrastive Learning (NeurIPS 2024 · similarity 0.89)
- Quantifying and Enhancing Multi-modal Robustness with Modality Preference (ICLR 2024 · similarity 0.88)
- Towards the Causal Complete Cause of Multi-Modal Representation Learning (ICML 2025 · similarity 0.87)
- ReconBoost: Boosting Can Achieve Modality Reconcilement (ICML 2024 · similarity 0.87)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.87)
- A Closer Look at Multimodal Representation Collapse (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
