Research map / Graph learning / Knowledge and temporal graphs
Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrum
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GS-MCC, a graph-spectrum framework that uses sliding-window multimodal interaction graphs, Fourier graph operators for low- and high-frequency filtering, and contrastive learning to collaborate the two frequency signals for multimodal emotion recognition in conversation.
Problem
Multimodal emotion recognition in conversation (MERC) with GNNs suffers from over-smoothing and low-pass filtering, failing to capture long-distance consistent (low-frequency) and complementary (high-frequency) multimodal information.
Approach
Revisit MERC through graph spectrum: Fourier graph operators with low-pass and high-pass filters extract long-distance low- and high-frequency node representations, and frequency-domain contrastive losses (LFCL/HFCL) make the two frequency signals collaborate.
Key results
- GS-MCC reaches 73.8% W-Acc and 73.9% W-F1 on IEMOCAP, 3.3% and 3.2% above AdaGIN.
- On MELD, GS-MCC reaches 68.1% W-Acc and 69.0% W-F1, outperforming AdaGIN by 0.5 and 2.2 points.
- GS-MCC has 2.10M parameters, far fewer than DialogueGCN (12.92M) and DER-GCN (78.59M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add speaker embedding (ablation) | GS-MCC · IEMOCAP and MELD | W-Acc and W-F1: IEMOCAP: 70.3 (↓2.8) W-Acc, 70.6 (↓2.7) W-F1; MELD: 65.4 (↓2.7) W-Acc, 64.6 (↓4.4) W-F1 | Helped |
| add Fourier GNN (ablation) | GS-MCC · IEMOCAP and MELD | W-Acc and W-F1: IEMOCAP: 68.7 (↓4.4) W-Acc, 67.7 (↓5.6) W-F1; MELD: 64.2 (↓3.9) W-Acc, 64.1 (↓4.9) W-F1 | Helped |
| add contrastive learning (ablation) | GS-MCC · IEMOCAP and MELD | W-Acc and W-F1: IEMOCAP: 70.3 (↓2.8) W-Acc, 71.3 (↓2.0) W-F1; MELD: 66.1 (↓2.0) W-Acc, 65.9 (↓3.1) W-F1 | Helped |
| use all three modalities rather than a single modality (ablation) | GS-MCC · IEMOCAP and MELD | W-Acc and W-F1: IEMOCAP: T+A+V 73.8 W-Acc, 73.9 W-F1; MELD: T+A+V 68.1 W-Acc, 69.0 W-F1 | Helped |
| add text modality to audio + vision (ablation) | GS-MCC · IEMOCAP and MELD | W-Acc and W-F1: IEMOCAP: 63.7 (↓10.1) -> 73.8 W-Acc, 63.0 (↓10.9) -> 73.9 W-F1; MELD: 54.6 (↓13.5) -> 68.1 W-Acc, 53.4 (↓15.6) -> 69.0 W | Helped |
| add audio modality to text + vision (ablation) | GS-MCC · IEMOCAP and MELD | W-Acc and W-F1: IEMOCAP: 69.5 (↓4.3) -> 73.8 W-Acc, 68.7 (↓5.2) -> 73.9 W-F1; MELD: 64.2 (↓3.9) -> 68.1 W-Acc, 64.1 (↓4.9) -> 69.0 W-F1 | Helped |
| add vision modality to text + audio (ablation) | GS-MCC · IEMOCAP and MELD | W-Acc and W-F1: IEMOCAP: 71.6 (↓2.2) -> 73.8 W-Acc, 71.0 (↓2.9) -> 73.9 W-F1; MELD: 66.3 (↓1.8) -> 68.1 W-Acc, 65.9 (↓3.1) -> 69.0 W-F1 | Helped |
| use GS-MCC instead of the strongest reported W-Acc baseline | MERC · IEMOCAP and MELD | W-Acc: IEMOCAP: 70.5 -> 73.8; MELD: 67.6 -> 68.1 | Helped |
Surprising findings
- Ablation shows removing the Fourier GNN hurts more than removing contrastive learning on both datasets.
- Video-only features perform worst, while text-only features are the strongest single modality.
Limitations
- Reported over-smoothing and efficiency claims rest on limited analysis; the O(N) vs O(N^2) argument is informal.
- Only two benchmark datasets (IEMOCAP and MELD) are evaluated.
- Some reported numbers are inconsistent across tables (e.g., ablation full-model W-Acc 73.1 vs main table 73.8).
Experimental setup
Datasets: IEMOCAP MELD
Benchmarks: IEMOCAP MELD
Metrics: accuracy F1 weighted accuracy weighted F1
Baselines: bc-LSTM LFM A-DMN DialogueGCN RGAT CoMPM EmoBERTa COGMEN CTNet LR-GCN MMGCN AdaGIN
Builds on: FourierGNN SpCo MMGCN M$^3$Net
Closest papers by meaning
- Adaptive Graph Learning for Multimodal Conversational Emotion Detection (AAAI 2024 · similarity 0.89)
- BIG-FUSION: Brain-Inspired Global-Local Context Fusion Framework for Multimodal Emotion Recognition in Conversations (AAAI 2025 · similarity 0.89)
- Dynamic Interactive Bimodal Hypergraph Networks for Emotion Recognition in Conversations (AAAI 2025 · similarity 0.86)
- Multi-to-Single: Reducing Multimodal Dependency in Emotion Recognition Through Contrastive Learning (AAAI 2025 · similarity 0.86)
- G^2SAM: Graph-Based Global Semantic Awareness Method for Multimodal Sarcasm Detection (AAAI 2024 · similarity 0.84)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.