Research map / Vision-language and multimodal / Multimodal LLMs
MERBench: A Unified Evaluation Benchmark for Multimodal Emotion Recognition
Zheng Lian, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, Jianhua Tao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use facial-expression-trained visual encoder (ResNet-FER2013) instead of ImageNet-trained | ResNet visual encoder · MER2023 | WAF (average): 44.39 -> 56.33 | Improved |
| use deep acoustic features (HUBERT-large) instead of handcrafted eGeMAPS | acoustic features · MER2023 | WAF (average): 19.62 -> 72.79 | Improved |
| train VideoMAE-base on human-centric MER-SEMI instead of Kinetics-400 (ablation) | VideoMAE-base · MER-MULTI | WAF: 46.72 -> 60.86 | Improved |
| train VideoMAE-base on VoxCeleb2 instead of Kinetics-400 (ablation) | VideoMAE-base · MER-MULTI | WAF: 46.72 -> 61.76 | Improved |
| train VideoMAE-large on VoxCeleb2 instead of Kinetics-400 (ablation) | VideoMAE-large · MER-MULTI | WAF: 53.26 -> 58.35 | Improved |
| match acoustic encoder language to input (Chinese HUBERT-large with Chinese audio rather than English) (ablation) | HUBERT-large · MER-MULTI | WAF: 29.52 -> 42.82 | Improved |
| feed Chinese instead of English TTS audio to English data2vec-base (ablation) | data2vec-base · MER-MULTI | WAF: 35.03 -> 28.61 | Hurt |
| translate Chinese text to English for Baichuan-13B (ablation) | Baichuan-13B · MER-MULTI | WAF: 62.54 -> 55.56 | Hurt |
| translate Chinese text to English for Llama2-13B (ablation) | Llama2-13B · MER-MULTI | WAF: 51.63 -> 53.77 | Improved |
| fuse three modalities instead of best unimodal feature | HUBERT-large, Baichuan-13B, CLIP-large · MER2023 | WAF (average): 72.79 -> 81.34 | Improved |
| use a high-performance feature set instead of medium-performance set with Attention fusion (ablation) | Attention fusion · Multimodal benchmark (mean of 8 datasets) | WAF: 74.91 -> 78.12 | Improved |
| add more top-ranked features per modality (top-1 to top-6) in attention fusion (ablation) | Attention fusion · Multimodal benchmark (mean of 8 datasets) | WAF: 78.12 -> 79.92 | Improved |
Closest papers by meaning
- OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition (ICML 2025 · similarity 0.91)
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.90)
- Multi-to-Single: Reducing Multimodal Dependency in Emotion Recognition Through Contrastive Learning (AAAI 2025 · similarity 0.88)
- Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.88)
- EMOE: Modality-Specific Enhanced Dynamic Emotion Experts (CVPR 2025 · similarity 0.87)
- AffectGPT-RL: Revealing Roles of Reinforcement Learning in Open-Vocabulary Emotion Recognition (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
