Research map / Vision-language and multimodal / Cross-modal learning
EMOE: Modality-Specific Enhanced Dynamic Emotion Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EMOE, a sample-adaptive mixture-of-modality-experts fusion with unimodal distillation for multimodal emotion recognition, also extended to intent recognition.
Problem
Multimodal emotion recognition suffers from a modality balance dilemma (modality importance varies per sample) and modality specialization disappearance (fusion favors invariant features and neglects unimodal predictive ability).
Approach
Treat each modality as an expert with a router network that learns per-sample modality weights for fusion, add router entropy and weight-importance similarity balancing losses, and distill unimodal predictions (weighted by router weights) into the multimodal prediction.
Key results
- On CMU-MOSI aligned setting, EMOE reaches 47.7% ACC7 and 85.4% ACC2, versus 46.2% ACC7 for DMD.
- On CMU-MOSEI aligned setting, EMOE achieves 54.1% ACC7 and 0.536 MAE, the best among compared methods.
- On MIntRec, EMOE improves precision to 72.08% over MISA's 71.24%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EMOE instead of DMD | CMU-MOSI aligned · CMU-MOSI aligned | ACC7: 46.2 -> 47.7 | Helped |
| use EMOE instead of DMD | CMU-MOSI unaligned · CMU-MOSI unaligned | ACC7: 46.7 -> 47.8 | Helped |
| use EMOE instead of Self-MM | CMU-MOSEI aligned · CMU-MOSEI aligned | ACC7: 53.2 -> 54.1 | Helped |
| use EMOE instead of MulT | CMU-MOSEI unaligned · CMU-MOSEI unaligned | ACC7: 53.2 -> 53.9 | Helped |
| use EMOE instead of MulT | MIntRec · MIntRec | ACC: 72.58 -> 72.58 | No clear effect |
| add Mixture of Modality Experts and Unimodal Distillation (ablation) | EMOE baseline · CMU-MOSI | ACC7: 43.0 -> 47.7 | Helped |
| add Unimodal Distillation (ablation) | EMOE with Mixture of Modality Experts · CMU-MOSI | ACC7: 46.2 -> 47.7 | Helped |
| add Mixture of Modality Experts (ablation) | EMOE with Unimodal Distillation · CMU-MOSI | ACC7: 45.0 -> 47.7 | Helped |
| add router entropy loss (ablation) | EMOE balancing mechanism · CMU-MOSI | ACC7: 45.3 -> 47.8 | Helped |
| add weight-importance similarity loss (ablation) | EMOE balancing mechanism · CMU-MOSI | ACC7: 46.8 -> 47.8 | Helped |
| replace summation fusion with concatenation fusion (ablation) | EMOE · CMU-MOSI aligned | ACC7: 47.7 -> 45.9 | Hurt |
| replace summation fusion with concatenation fusion (ablation) | EMOE · CMU-MOSI unaligned | ACC7: 47.8 -> 47.4 | Hurt |
Surprising findings
- Concatenation fusion is close to summation fusion in performance, with the unaligned concatenation variant matching summation on several metrics.
Limitations
- Gains over prior methods are small on several metrics and some results are comparable rather than superior.
- Hyperparameter sensitivity analysis excludes very low temperature settings that cause unstable router initialization.
Experimental setup
Datasets: CMU-MOSI CMU-MOSEI MIntRec
Benchmarks: CMU-MOSI CMU-MOSEI MIntRec
Metrics: ACC7 ACC2 F1 MAE ACC Precision Recall
Baselines: EF-LSTM LF-DNN TFN LMF MFN Graph-MFN MCTN MulT MISA Self-MM MMIM DMD
Builds on: Mixture of Experts MISA DMD one-way knowledge distillation
Closest papers by meaning
- Multi-to-Single: Reducing Multimodal Dependency in Emotion Recognition Through Contrastive Learning (AAAI 2025 · similarity 0.90)
- OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition (ICML 2025 · similarity 0.88)
- MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.87)
- MODA: MOdular Duplex Attention for Multimodal Perception, Cognition, and Emotion Understanding (ICML 2025 · similarity 0.87)
- Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.87)
- Nano-EmoX: Unifying Multimodal Emotional Intelligence from Perception to Empathy (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.