Research map / Vision-language and multimodal / Cross-modal learning

A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities

AAAI 2024 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal sentiment analysis · 59 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UMDF, a single-network self-distillation framework with multi-grained crossmodal attention and dynamic feature integration for multimodal sentiment analysis under uncertain missing modalities.

Problem

Multimodal sentiment analysis degrades when modalities are missing at random during training and testing, since fixed fusion methods and fixed missing-case training do not generalize to stochastic real-world missingness.

Approach

Generate two heterogeneous missing-modality versions of each sample and distill feature (MMD) and logit (JS or MSE) distributions bidirectionally between them within one shared network, combined with coarse-to-fine crossmodal attention and a gated frame-level feature integration module.

Key results

• On MOSI with all modalities, UMDF reaches 0.782 MAE and 83.36 F1 in the full-modality setting, while averaging 75.56 F1 across six missing-modality test conditions (vs. 74.00-ish range for baselines such as GCNet at 73.84).
• On MOSEI, UMDF averages 74.24 F1 across missing-modality conditions, outperforming GCNet (73.54) and SMIL (70.65).
• On IEMOCAP, UMDF achieves the highest F1 in all four emotion categories under the average missing-modality condition (e.g., 76.2 happy, 59.3 neutral).

Measured effects reported in the paper

Surprising findings

• Under only-language input with 0.5 missing rate, the authors note that the language+audio combination in some settings even outperforms complete-modality input on individual metrics, which they attribute to language carrying the most sentiment clues.

Limitations

• Missing modalities simulated by random dropping of frames or whole modalities, which may not match real-world missingness patterns.
• Results reported as averages over five seeds; the paper does not discuss variance or statistical significance in detail.
• Baselines were re-implemented by the authors, which may affect comparability.

Experimental setup

Datasets: MOSI MOSEI IEMOCAP

Benchmarks: MOSI MOSEI IEMOCAP

Metrics: MAE F1 score accuracy

Baselines: Self-MM CubeMLP MCTN TransM SMIL GCNet

Builds on: knowledge distillation Self-MM MMD crossmodal transformer

Closest papers by meaning

• Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities (CVPR 2024 · similarity 0.93)
• Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning (NeurIPS 2024 · similarity 0.93)
• DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.91)
• Towards Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2024 · similarity 0.91)
• A Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.90)
• SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2026 · similarity 0.90)