Research map / Vision-language and multimodal / Cross-modal learning
Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CorrKD, a teacher-student knowledge distillation framework for multimodal sentiment analysis under intra- and inter-modality missingness, using sample-level contrastive, category-prototype, and response-disentangled distillation.
Problem
Multimodal sentiment analysis degrades severely when modalities (frames or entire modalities) are missing at test time, and prior missing-modality methods ignore cross-sample, cross-category, and response-level correlations.
Approach
Distill from a complete-modality teacher to a missing-modality student using three losses: contrastive distillation across samples in a batch, category-prototype similarity distillation to align feature distributions, and disentangled target/non-target response distillation maximizing mutual information (via Jensen-Shannon estimate).
Key results
- On MOSI, CorrKD average F1 under six inter-modality missing conditions is 74.69 vs 73.84 for GCNet (complete-modality 83.94).
- On MOSI, CorrKD improves F1 by 3.72% over GCNet in the {a, v} testing condition (language missing).
- On MOSEI, CorrKD average F1 is 74.02 vs 73.54 for GCNet under inter-modality missingness.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CorrKD instead of GCNet | MOSI · inter-modality missingness (Avg.) | F1 score: 73.84 -> 74.69 | Helped |
| Use CorrKD instead of GCNet | MOSI · complete modalities | F1 score: 83.20 -> 83.94 | Helped |
| Use CorrKD instead of GCNet | MOSEI · inter-modality missingness (Avg.) | F1 score: 73.54 -> 74.02 | Helped |
| Use CorrKD instead of the strongest complete-modality baseline, DMD | MOSEI · complete modalities | F1 score: 84.78 -> 82.16 | Hurt |
| Use CorrKD instead of GCNet | IEMOCAP · happy, inter-modality missingness (Avg.) | F1 score: 75.3 -> 76.1 | Helped |
| Use CorrKD instead of GCNet | IEMOCAP · sad, inter-modality missingness (Avg.) | F1 score: 75.4 -> 76.6 | Helped |
| Use CorrKD instead of GCNet | IEMOCAP · angry, inter-modality missingness (Avg.) | F1 score: 73.8 -> 74.8 | Helped |
| Use CorrKD instead of GCNet | IEMOCAP · neutral, inter-modality missingness (Avg.) | F1 score: 57.6 -> 59.9 | Helped |
| Add sample-level contrastive distillation (SCD) (ablation) | CorrKD · MOSI, inter-modality missingness (Avg.) | F1 score: 72.46 -> 74.69 | Helped |
| Add category-guided prototype distillation (CPD) (ablation) | CorrKD · MOSI, inter-modality missingness (Avg.) | F1 score: 71.81 -> 74.69 | Helped |
| Add response-disentangled consistency distillation (RCD) (ablation) | CorrKD · MOSI, inter-modality missingness (Avg.) | F1 score: 73.27 -> 74.69 | Helped |
| Replace the proposed RCD with KL divergence loss (ablation) | CorrKD · MOSI, missing-modality scenarios | F1 score | Hurt |
Surprising findings
- Language-only inputs with CorrKD perform comparably to complete-modality testing, and bimodal settings with language sometimes exceed complete-modality scores on individual metrics.
Limitations
- Gains over GCNet are modest on several metrics (e.g., MOSEI complete-modality and some MOSI conditions where CorrKD is lower).
- Baselines were re-implemented by the authors using public codebases, so comparisons depend on their reproduction.
Experimental setup
Datasets: MOSI MOSEI IEMOCAP
Benchmarks: MOSI MOSEI IEMOCAP
Metrics: weighted F1 F1 score per emotion category
Baselines: Self-MM CubeMLP DMD MCTN TransM SMIL GCNet
Builds on: knowledge distillation MCTN Self-MM
Closest papers by meaning
- A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities (AAAI 2024 · similarity 0.93)
- Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning (NeurIPS 2024 · similarity 0.90)
- DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.89)
- Towards Multimodal Sentiment Analysis via Hierarchical Correlation Modeling with Semantic Distribution Constraints (AAAI 2025 · similarity 0.89)
- SemMSA: Latent Semantic-Aided Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2026 · similarity 0.88)
- Towards Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.