Research map / Vision-language and multimodal / Cross-modal learning
Towards Robust Multimodal Sentiment Analysis with Incomplete Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LNLN, a language-dominated noise-resistant network that corrects the dominant (language) modality via completeness check and adversarial proxy feature generation, and benchmarks robustness of MSA methods under random data missing on MOSI, MOSEI and SIMS.
Problem
Multimodal sentiment analysis models degrade under incomplete data (random modality missing), and prior evaluations use inconsistent metrics and settings.
Approach
Treat language as the dominant modality and improve its representation under noise: a Dominant Modality Correction module predicts data completeness and adversarially generates a proxy dominant feature that is blended with the language feature, followed by hyper-modality learning with visual/audio features and a reconstructor for missing inputs.
Key results
- On MOSI, LNLN reaches 34.26 Acc-7, a 9.46% relative improvement over the sub-optimal MMIM.
- On SIMS, LNLN achieves F1 79.43, a 9.17% relative improvement over ALMT.
- On MOSEI, LNLN is only sub-optimal on Acc-7 and Acc-5 (CENET is best), though it leads on several other metrics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LNLN instead of the strongest baseline shown, MMIM | MOSI · MOSI | Acc-7: 31.30 -> 34.26 | Helped |
| Use LNLN instead of CENET | MOSEI · MOSEI | Acc-7: 47.18 -> 45.42 | Hurt |
| Use LNLN instead of ALMT | MOSEI · MOSEI | MAE: 0.674 -> 0.692 | Hurt |
| Use LNLN instead of ALMT | SIMS · SIMS | F1: 72.76 -> 79.43 | Helped |
| Add DMC and reconstructor (ablation) | LNLN · MOSI | Acc-7: 29.54 -> 34.26 | Helped |
| Add DMC (ablation) | LNLN · MOSI | Acc-7: 33.94 -> 34.26 | Helped |
| Add reconstructor (ablation) | LNLN · MOSI | Acc-7: 33.94 -> 34.26 | Helped |
| Use DMML instead of concatenation fusion (ablation) | LNLN · MOSI | Acc-7: 33.22 -> 34.26 | Helped |
| Train with missing data (ablation) | LNLN · MOSI | Acc-7: 30.11 -> 34.26 | Helped |
| Add completeness-check loss (ablation) | LNLN · MOSI | Acc-7: 33.61 -> 34.26 | Helped |
| Add adversarial loss (ablation) | LNLN · MOSI | Acc-7: 32.84 -> 34.26 | Helped |
| Add reconstructor loss (ablation) | LNLN · MOSI | Acc-5: 38.52 -> 38.27 | Hurt |
Surprising findings
- Many methods converge to similar Acc-7/Acc-5 at high missing rates, indicating lazy predictions biased to majority classes.
- Removing the DMC and reconstructor individually causes only slight drops, while removing both causes a larger drop.
Limitations
- Not always the best in modality-missing scenarios, indicating limited multi-scene generalization.
- Hyperparameter tuning of loss weights is challenging and sensitive.
- Does not address other real-world noise factors such as cultural context or platform-specific features.
Experimental setup
Datasets: MOSI MOSEI SIMS
Benchmarks: MOSI MOSEI SIMS
Metrics: Acc-7 Acc-5 Acc-3 Acc-2 F1 MAE Corr
Baselines: MISA Self-MM MMIM CENET TETFN TFR-Net ALMT
Builds on: ALMT MMSA Self-MM gradient reversal layer
Closest papers by meaning
- Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning (NeurIPS 2024 · similarity 0.91)
- A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities (AAAI 2024 · similarity 0.91)
- Gradient-Guided Modality Decoupling for Missing-Modality Robustness (AAAI 2024 · similarity 0.89)
- SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2026 · similarity 0.88)
- DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.88)
- A Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.