Research map / Vision-language and multimodal / Cross-modal learning
MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MIDAS, a data augmentation that builds misaligned multimodal samples by swapping modalities across labels, labels them via unimodal confidence, and reweights them with weak-modality and hard-sample weights to counter modality imbalance.
Problem
Multimodal models over-rely on dominant modalities and underuse weaker ones, and prior fixes focus on training objectives or masking rather than data augmentation.
Approach
Generate misaligned samples by pairing modalities from different-label instances, supervise them with soft labels weighted by unimodal confidence, upweight the least-confident modality's loss (weak-modality weighting), and upweight misaligned samples whose swapped feature is cosine-similar to the original (hard-sample weighting).
Key results
- MIDAS reaches 74.88% accuracy on Kinetics-Sounds vs 71.75% for the best baseline (MCR) and 63.92% for joint training.
- On CREMA-D MIDAS achieves 74.99% accuracy, about 4.08 points above the best aligned-only baseline (AMCo, 69.91%).
- On UCF-101 MIDAS (95.20%) is roughly tied with SMV (95.24%), which uses about 3x more generated data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with MIDAS misaligned samples and weighting | Multimodal classification models · Kinetics-Sounds | Acc: 71.75 ± 0.56 -> 74.88 ± 0.49 | Helped |
| Train with MIDAS misaligned samples and weighting | Multimodal classification models · CREMA-D | Acc: 70.91 ± 1.22 -> 74.99 ± 0.31 | Helped |
| Train with MIDAS misaligned samples and weighting | Multimodal classification models · UCF-101 | Acc: 95.24 ± 0.39 -> 95.20 ± 0.18 | No clear effect |
| Train with MIDAS misaligned samples and weighting | Multimodal classification models · Food-101 | Acc: 92.40 ± 0.27 -> 93.46 ± 0.36 | Helped |
| Add warm-up training for unimodal classifiers (ablation) | MIDAS · Kinetics-Sounds | Acc: 71.70 ± 0.61 -> 72.32 ± 0.52 | Helped |
| Add weak-modality weighting (ablation) | MIDAS · Kinetics-Sounds | Acc: 71.70 ± 0.61 -> 72.84 ± 0.41 | Helped |
| Add hard-sample weighting (ablation) | MIDAS · Food-101 | Acc: 93.39 ± 0.19 -> 93.51 ± 0.29 | No clear effect |
| Use all three components: warm-up, weak-modality weighting, and hard-sample weighting (ablation) | MIDAS · CREMA-D | Acc: 72.32 ± 0.52 -> 74.99 ± 0.31 | Helped |
| Replace cosine similarity with L2 distance for hard-sample weighting (ablation) | MIDAS · Kinetics-Sounds | Acc: 75.26 -> 74.69 | Hurt |
| Replace random replacement with confusion-based replacement (ablation) | MIDAS · CREMA-D | Acc: 75.00% -> 67.67% | Hurt |
| Set misaligned-sample loss weight λ to 5 (ablation) | MIDAS · Food-101 | Acc: 93.82 -> 65.20 | Hurt |
| Increase λ from 1 to 10 (ablation) | MIDAS · CREMA-D | Acc: 75.00 -> 63.17 | Hurt |
Surprising findings
- Confusion-based pairing of misaligned samples (top-2 predicted labels) reduced CREMA-D accuracy to 67.67% versus 75.00% for random replacement.
- Mixup, PowMix and LeMDA sometimes underperform joint training on CREMA-D, with PowMix and LeMDA below joint training on some metrics.
Limitations
- Evaluated only on classification tasks; extension to generation or retrieval needs adaptation.
- Hyperparameter lambda is dataset-dependent (e.g., best 5 on Kinetics-Sounds, 1 on CREMA-D and Food-101); large lambda degrades performance.
Experimental setup
Datasets: Kinetics-Sounds CREMA-D UCF-101 Food-101 CMU-MOSI Sarcasm Twitter2015
Benchmarks: Kinetics-Sounds CREMA-D UCF-101 Food-101 CMU-MOSI Sarcasm Twitter2015
Metrics: Accuracy F1-score
Baselines: Joint training SMV OPM AMCo LFM MCR Mixup PowMix LeMDA
Builds on: OGM-GE AGM MCR LFM Mixup
Closest papers by meaning
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.90)
- Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion (NeurIPS 2025 · similarity 0.89)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.89)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.88)
- UNIALIGN: Scaling Multimodal Alignment within One Unified Model (CVPR 2025 · similarity 0.88)
- Beyond Modality Fusion: Deep Ensembles for Multimodal Classification (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.