Research map / Graph learning / Graph self-supervised learning
Adaptive Graph Learning for Multimodal Conversational Emotion Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaIGN, a directed adaptive interactive graph network that uses Gumbel-Softmax node/edge/graph-level selection policies and a keep-or-drop loss to model intra- and cross-modal dependencies for multimodal emotion recognition in conversations.
Problem
Multimodal emotion recognition in conversations must balance intra- and inter-speaker context (emotional inertia vs. empathy) and handle cross-modal conflicts, without using future utterances.
Approach
A directed interactive graph with learnable Gumbel-Softmax node-, edge-, and graph-level selection policies that adaptively keep or drop nodes and edges, with a task-specific keep-or-drop loss prioritizing the text modality and intra-speaker context, and pseudo-labels from self-supervised sub-models to mask unnecessary nodes.
Key results
- AdaIGN reaches 70.74 weighted F1 on IEMOCAP and 66.79 on MELD, outperforming CMCF-SRNet (62.30 on IEMOCAP) and other baselines.
- Removing the selection policies drops IEMOCAP F1 by 3.86% and MELD by 2.76%.
- Removing the textual modality reduces IGN F1 by about 19.6% on IEMOCAP and 20.4% on MELD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AdaIGN's adaptive graph learning and selection policies | multimodal ERC · IEMOCAP | weighted F1: 69.60 -> 70.74 | Helped |
| add AdaIGN's adaptive graph learning and selection policies | multimodal ERC · MELD | weighted F1: 62.30 -> 66.79 | Helped |
| add acoustic modality (ablation) | IGN · IEMOCAP | weighted F1: 65.97 -> 66.88 | Helped |
| add visual modality (ablation) | IGN · IEMOCAP | weighted F1: 66.10 -> 66.88 | Helped |
| add text modality (ablation) | IGN · IEMOCAP | weighted F1: 47.25 -> 66.88 | Helped |
| add intra-speaker context (ablation) | IGN · IEMOCAP | weighted F1: 65.64 -> 66.88 | Helped |
| add inter-speaker context (ablation) | IGN · IEMOCAP | weighted F1: 66.32 -> 66.88 | Helped |
| add NSP (ablation) | AdaIGN · IEMOCAP | weighted F1: 68.02 -> 70.74 | Helped |
| add ESP (ablation) | AdaIGN · MELD | weighted F1: 64.96 -> 66.79 | Helped |
| add GSP (ablation) | AdaIGN · MELD | weighted F1: 65.39 -> 66.79 | Helped |
| add Lk (ablation) | AdaIGN · IEMOCAP | weighted F1: 67.16 -> 70.74 | Helped |
| add Lm (ablation) | AdaIGN · MELD | weighted F1: 65.76 -> 66.79 | Helped |
Surprising findings
- Conflicting emotion predictions across modalities are common, with conflict ratios up to 83.73% for acoustic-visual on IEMOCAP.
- Performance on utterances with emotional shifts is substantially lower than without shifts (58.83 vs 75.93 weighted F1 on IEMOCAP).
Limitations
- Performance drops substantially on emotional-shift utterances.
- Class imbalance hurts minority classes such as fear (F1 15.15 on MELD).
- Relies on pseudo-labels from auxiliary sub-models to reduce policy training cost.
Experimental setup
Datasets: IEMOCAP MELD
Benchmarks: IEMOCAP MELD
Metrics: weighted F1 accuracy per-class F1
Baselines: DialogueRNN DialogueGCN CTNet MMGCN MMDFN SCMM CMCF-SRNet
Builds on: MMGCN DialogueRNN GCNII Gumbel-Softmax
Closest papers by meaning
- Dynamic Interactive Bimodal Hypergraph Networks for Emotion Recognition in Conversations (AAAI 2025 · similarity 0.90)
- Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrum (AAAI 2025 · similarity 0.89)
- BIG-FUSION: Brain-Inspired Global-Local Context Fusion Framework for Multimodal Emotion Recognition in Conversations (AAAI 2025 · similarity 0.89)
- Utterance-level Emotion Recognition in Conversation with Conversation-level Supervision (AAAI 2025 · similarity 0.88)
- Emotion Rendering for Conversational Speech Synthesis with Heterogeneous Graph-Based Context Modeling (AAAI 2024 · similarity 0.87)
- Dual-View Learning for Conversational Emotion Recognition Through Context and Emotion-Shift Modeling (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.