Research map / Vision-language and multimodal / Cross-modal learning
DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DLF, a disentangled-language-focused framework for multimodal sentiment analysis that separates shared and specific features with four geometric regularizers and enhances the language modality via a language-focused attractor with hierarchical predictions.
Problem
Multimodal sentiment analysis suffers from redundancy and conflicts when all modalities are treated equally or when information is transferred bidirectionally between all modality pairs.
Approach
Disentangle modality-shared and modality-specific features with geometric regularizers (reconstruction, specific, modified triplet, soft orthogonality), then use a language-query cross-attention attractor to transfer complementary information only into the dominant language modality, plus hierarchical predictions over shared, specific, and fused outputs.
Key results
- On CMU-MOSI, DLF reaches 47.08 Acc-7 and 85.06 Acc-2, the best among compared methods in the table.
- On CMU-MOSEI, DLF reaches 53.90 Acc-7, 85.42 Acc-2 and 0.536 MAE, outperforming DMD (52.78 Acc-7) and MAG-BERT.
- Ablation on MOSI: removing LFA drops Acc-7 from 47.08 to 42.71; removing hierarchical predictions drops it to 42.42.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the feature disentanglement module (ablation) | DLF · CMU-MOSI | Acc-7: 45.92 -> 47.08 | Helped |
| Add the Language-Focused Attractor instead of three separate Query components (ablation) | DLF · CMU-MOSI | Acc-7: 42.71 -> 47.08 | Helped |
| Add hierarchical predictions (ablation) | DLF · CMU-MOSI | Acc-7: 42.42 -> 47.08 | Helped |
| Add reconstruction loss L_r (ablation) | DLF · CMU-MOSI | Acc-7: 45.92 -> 47.08 | Helped |
| Add specific loss L_s (ablation) | DLF · CMU-MOSI | Acc-7: 45.36 -> 47.08 | Helped |
| Add modified triplet loss L_m (ablation) | DLF · CMU-MOSI | Acc-7: 45.77 -> 47.08 | Helped |
| Add soft orthogonality loss L_o (ablation) | DLF · CMU-MOSI | Acc-7: 45.77 -> 47.08 | Helped |
| Use language and audio modalities rather than language alone (ablation) | DLF · CMU-MOSI | Acc-7: 45.63 -> 45.77 | Helped |
| Use language and vision modalities rather than language alone (ablation) | DLF · CMU-MOSI | Acc-7: 45.63 -> 46.65 | Helped |
| Use only audio modality instead of the full multimodal input (ablation) | DLF · CMU-MOSI | Acc-7: 47.08 -> 15.31 | Hurt |
| Use only vision modality instead of the full multimodal input (ablation) | DLF · CMU-MOSI | Acc-7: 47.08 -> 15.01 | Hurt |
Surprising findings
- Language alone (only L) achieves 45.63 Acc-7 on MOSI, close to the full model, while audio-only and vision-only are near chance-level.
Limitations
- Only considers complete-modality scenarios; missing modalities limit the disentanglement and enhancement modules.
- Performance is limited on under-represented sentiment classes (HN and HP) due to long-tailed data distribution.
Experimental setup
Datasets: CMU-MOSI CMU-MOSEI
Benchmarks: CMU-MOSI CMU-MOSEI
Metrics: Acc-7 Acc-5 Acc-2 F1 Corr MAE
Baselines: EF-LSTM LF-DNN TFN LMF MFN Graph-MFN MulT PMR MISA MAG-BERT DMD
Builds on: MulT MISA DMD
Closest papers by meaning
- Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning (NeurIPS 2024 · similarity 0.91)
- A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities (AAAI 2024 · similarity 0.91)
- Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities (CVPR 2024 · similarity 0.89)
- Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio Content (AAAI 2025 · similarity 0.88)
- A Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.88)
- Towards Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.