Research map / Vision-language and multimodal / Cross-modal learning
Enriching Multimodal Sentiment Analysis Through Textual Emotional Descriptions of Visual-Audio Content
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DEVA, a multimodal sentiment analysis framework that converts audio prosody and facial action units into textual emotional descriptions and fuses them progressively under text guidance, reporting state-of-the-art results on MOSI, MOSEI, and CH-SIMS.
Problem
Multimodal sentiment analysis struggles to discern subtle emotional intensity differences in audio and visual segments whose polarities look similar, and fusing heterogeneous modalities suffers from distribution gaps between text and visual-audio features.
Approach
Textualize audio (OpenSMILE prosody tertile levels) and visual (OpenFace action units) cues into natural-language emotional descriptions, concatenate them with source features, and progressively fuse audio/visual minor modalities into text-guided representations via cross-modal attention.
Key results
- DEVA reaches 84.40/86.29 Acc-2 on MOSI and 83.26/86.13 Acc-2 on MOSEI, the best reported in the comparison table.
- On CH-SIMS, DEVA achieves Acc-2 79.64, Acc-3 65.42, Acc-5 43.07 and F1 80.32, outperforming the best baseline on these metrics.
- Ablation shows removing the Minor-modality Fusion Unit causes the largest drop (MOSI Acc-5 falls from 50.44 to 23.29).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add emotional descriptions (AED and VED) (ablation) | DEVA · MOSI | Acc-7: 43.00 -> 46.21 | Helped |
| Add emotional descriptions (AED and VED) (ablation) | DEVA · CH-SIMS | Acc-2: 77.89 -> 79.64 | Helped |
| Add VED (ablation) | DEVA · MOSI | Acc-7: 40.96 -> 46.21 | Helped |
| Add AED (ablation) | DEVA · MOSI | Acc-7: 45.77 -> 46.21 | Helped |
| Replace audio and visual source data with AED and VED (ablation) | DEVA · CH-SIMS | Acc-2: 79.86; 79.64 | No clear effect |
| Add CEU (ablation) | DEVA · MOSI | Acc-5: 48.98 -> 50.44 | Helped |
| Add MFU (ablation) | DEVA · MOSI | Acc-5: 23.29 -> 50.44 | Helped |
| Add EDG (ablation) | DEVA · CH-SIMS | Acc-2: 77.89 -> 79.64 | Helped |
| Use crossmodal Fusion Layer instead of simple concatenation (ablation) | DEVA · CH-SIMS | Acc-2: 77.46 -> 79.64 | Helped |
| Encode VED with BERT instead of one-hot vectors (ablation) | DEVA · MOSI | Acc-5: 47.67 -> 50.44 | Helped |
| Set α=1.0, β=1.0 rather than the tested alternatives (ablation) | DEVA · MOSI | Acc-2: 85.21; 84.91; 84.6; 85.37; 86.28 | Mixed |
| Use the prompt “The speaker made such an expression” rather than the tested alternative prompts (ablation) | DEVA · MOSI | Acc-2: 84.4/86.28; 83.36/85.6; 83.67/85.98 | No clear effect |
Surprising findings
- Replacing raw audio and visual inputs with only textual emotional descriptions (DEVA w/o AV) keeps performance comparable to the full model on MOSI and CH-SIMS.
- Adding visual emotional descriptions gives a larger gain than adding audio emotional descriptions.
Limitations
- Emotional descriptions are not fluent natural language, as the authors acknowledge.
- Gains over baselines are small on several metrics (e.g., MOSI MAE and Corr are not best; MOSEI Acc-7 and CH-SIMS Corr trail), and results are from a single reported setup.
Experimental setup
Datasets: MOSI MOSEI CH-SIMS
Benchmarks: MOSI MOSEI CH-SIMS
Metrics: Acc-2 Acc-5 Acc-7 Acc-3 F1 MAE Corr
Baselines: TFN LMF MulT ICCN MISA MAG-BERT PMR MFSA FDMER Self-MM ALMT ConFEDE
Builds on: BERT Self-MM ALMT MulT OpenFace
Closest papers by meaning
- DLF: Disentangled-Language-Focused Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.88)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.88)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.88)
- EmoThinker: Advancing Visual-Acoustic Emotion Analysis via Structural Token Selection and Chain-of-Thought Reasoning (CVPR 2026 · similarity 0.86)
- Uncertain Multimodal Intention and Emotion Understanding in the Wild (CVPR 2025 · similarity 0.86)
- A Multi-Focus-Driven Multi-Branch Network for Robust Multimodal Sentiment Analysis (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.