Research map / Vision-language and multimodal / Cross-modal learning

Towards Multimodal Sentiment Analysis via Hierarchical Correlation Modeling with Semantic Distribution Constraints

AAAI 2025 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal sentiment analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HCMNet, a multimodal sentiment analysis model that combines atomic-level cross-attention correlations and graph-based composition-level reasoning, with a contrastive loss and graph optimal transport to constrain semantic distribution differences across modalities.

Problem

Multimodal sentiment analysis mostly models atomic-level correlations between text, video and audio, ignoring composition-level (weak but vital) correlations and the semantic distribution inconsistency between modalities.

Approach

Jointly models atomic-level correlations via dynamic attention reasoning and composition-level correlations via GCNs over per-modality graphs, then constrains cross-modal distributions at both levels: an atomic-level cosine contrastive loss and a graph optimal transport module that fuses transport flows with video, audio and text graphs.

Key results

• HCMNet reaches 74.62 w-F1 on IEMOCAP and 68.94 w-F1 on MELD, 2.44 and 3.30 points above MultiEMO.
• On CMU-MOSEI, HCMNet achieves MAE 0.489, Corr 0.819, Acc-2 87.63 and F1 87.56, the best among compared methods.
• Ablation shows removing composition-level distribution constraint drops IEMOCAP w-F1 from 74.62 to 69.52.

Measured effects reported in the paper

Surprising findings

• Performance on MELD F1 is much lower than on IEMOCAP, attributed to IEMOCAP's scripted, emotionally clear utterances.
• On CMU-MOSEI the improvement in accuracy and F1 is only mediocre, as the authors acknowledge.

Limitations

• Gains on CMU-MOSEI accuracy and F1 are modest, indicating room for improvement.
• Future plan to incorporate large vision-language model knowledge is not yet realized.
• Hyperparameters (GCN layers, PGF layers) require tuning; performance declines beyond the chosen depth.

Experimental setup

Datasets: IEMOCAP MELD CMU-MOSEI

Benchmarks: IEMOCAP MELD CMU-MOSEI

Metrics: weighted-F1 accuracy MAE Pearson correlation F1

Baselines: DialogueRNN DialogueGCN ICCN IterativeERC MMIM MMGCN UniMSE i-Code MultiEMO AdaIGN

Builds on: UniMSE MultiEMO CLIP MMGCN

Closest papers by meaning

• Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities (CVPR 2024 · similarity 0.89)
• DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.87)
• Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning (NeurIPS 2024 · similarity 0.87)
• A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities (AAAI 2024 · similarity 0.87)
• A Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.87)
• SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2026 · similarity 0.87)