Research map / Vision-language and multimodal / Cross-modal learning

Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning

NeurIPS 2024 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal sentiment analysis · 9 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HRLF, a teacher-student framework that factorizes each modality into sentiment-relevant and modality-specific parts, then aligns multi-scale representations via hierarchical mutual information and adversarial learning to handle intra- and inter-modality missingness in multimodal sentiment analysis.

Problem

Multimodal sentiment analysis degrades when modalities are missing, either frame-level within sequences or entire modalities, and prior methods extract sentiment semantics poorly and align representations weakly.

Approach

Factorize each modality into sentiment-relevant and modality-specific representations via intra- and inter-modality translation plus sentiment semantic reconstruction, then distill from a complete-modality teacher to a missing-modality student using hierarchical mutual information maximization (JSD-based) and multi-scale adversarial alignment.

Key results

• On MOSI, HRLF averages 76.74 F1 across six inter-modality missing conditions vs 74.69 for CorrKD (p<0.05).
• On MOSEI, HRLF averages 75.63 vs 74.02 for CorrKD under inter-modality missingness.
• On IEMOCAP, HRLF reaches 78.5 F1 on happy and 61.7 on neutral in the averaged missing conditions, exceeding CorrKD.

Measured effects reported in the paper

Surprising findings

• Under language-only testing, HRLF performance approaches complete-modality performance, suggesting language dominates sentiment inference.
• Several complete-modality baselines (Self-MM, CubeMLP, DMD) collapse sharply under missing modalities, though they are strongest on the full-modality condition.

Limitations

• Missing-modality simulation may not capture the complexity of real-world missingness, so performance may drop in practice.
• Baselines were re-implemented by the authors under their own experimental paradigm, which may affect fair comparison.
• No theoretical analysis is provided for the mutual information and adversarial objectives.

Experimental setup

Datasets: MOSI MOSEI IEMOCAP

Benchmarks: MOSI MOSEI IEMOCAP

Metrics: MAE F1 F1 score per emotion class

Baselines: Self-MM CubeMLP DMD MCTN TransM SMIL GCNet CorrKD

Builds on: Deep InfoMax knowledge distillation CorrKD SMIL

Closest papers by meaning

• A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities (AAAI 2024 · similarity 0.93)
• DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.91)
• Towards Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2024 · similarity 0.91)
• SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2026 · similarity 0.90)
• Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities (CVPR 2024 · similarity 0.90)
• A Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.89)