Research map / Vision-language and multimodal / Human and embodied modeling
OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces open-vocabulary multimodal emotion recognition (OV-MER), builds the OV-MERD dataset via human-LLM collaboration, proposes set-based grouping metrics, and benchmarks zero-shot MLLMs on emotion-clue-based label prediction.
Problem
Multimodal emotion recognition relies on fixed, basic-emotion label spaces that fail to capture the nuance and multi-appraisal nature of human emotions; there is no dataset, metric, or solution for predicting arbitrary emotion labels.
Approach
Reformulate MER as open-vocabulary generation: annotate videos with arbitrary numbers of emotion labels extracted from LLM-merged multimodal clues, and evaluate with set-based precision/recall/F after grouping synonyms via GPT-based or emotion-wheel-based grouping.
Key results
- GPT-4V achieves the highest English F_s of 55.51 among CLUE-MLLM baselines, while CLUE-Multi (human-checked clues) reaches 80.05 F_s, showing a large gap for current MLLMs.
- EW-based grouping metrics correlate strongly with GPT-based metrics (PCC up to 0.942), suggesting EW can replace GPT grouping for cheaper, reproducible evaluation.
- In a user study, 97.5% of annotations judged OV-MERD labels more informative than basic emotion labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine checked visual and acoustic clues with text instead of using only checked acoustic clues | CLUE-Multi on OV-MERD · OV-MERD English | F_s: 65.35$_{\pm0.04}$ -> 80.05$_{\pm0.24}$ | Helped |
| Combine checked visual and acoustic clues with text instead of using only checked acoustic clues | CLUE-Multi on OV-MERD · OV-MERD Chinese | F_s: 68.56$_{\pm0.07}$ -> 85.16$_{\pm0.03}$ | Helped |
| Use checked audio clues with text rather than text only | CLUE-Audio on OV-MERD · OV-MERD English | F_s: 46.00$_{\pm0.06}$ -> 65.35$_{\pm0.04}$ | Helped |
| Use checked video clues rather than text only | CLUE-Video on OV-MERD · OV-MERD English | F_s: 46.00$_{\pm0.06}$ -> 60.55$_{\pm0.13}$ | Helped |
| Include text alongside video input to the MLLM (S1 rather than S0) (ablation) | CLUE-MLLM · OV-MERD | F_s | Helped |
| Extract descriptions with an MLLM, then combine them with text using another LLM (S2 rather than S1) (ablation) | CLUE-MLLM · OV-MERD | F_s | Helped |
| Use one-hot labels instead of open-vocabulary labels | OV-MERD label evaluation · OV-MERD English | Recall_s: 51.05$_{\pm0.08}$ | Hurt |
| Use human-LLM collaboration rather than human-only annotation | OV-MERD annotation · OV-MERD | informativeness preference: 97.5% | Helped |
| Use OV-MERD labels rather than basic emotion labels | OV-MERD annotation · OV-MERD | human-perception alignment: 96% | Helped |
Surprising findings
- GPT-based and matching-based metrics (BLEU, METEOR, ROUGE) correlate only weakly, with the highest PCC about 0.77.
- Feeding both video and text into an MLLM at once performs worse than extracting descriptions first and merging with text using a separate LLM.
Limitations
- Dataset derived from MER2023 movies/TV clips with mostly single-person videos, limiting spontaneity and generalization.
- Only a subset of MLLMs evaluated; cultural differences not studied (data originally Chinese).
- Ground truth partly depends on GPT-3.5 label extraction and GPT-based grouping, with reproducibility affected by API versions.
Experimental setup
Datasets: MER2023 OV-MERD
Benchmarks: OV-MERD
Metrics: Precision_s Recall_s F_s PCC
Baselines: Qwen-Audio OneLLM Otter Video-LLaMA VideoChat SECap PandaGPT Video-LLaVA SALMONN VideoChat2 Video-ChatGPT LLaMA-VID
Builds on: MER2023 SALMONN GPT-4V Emotion Wheel
Closest papers by meaning
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.92)
- AffectGPT-RL: Revealing Roles of Reinforcement Learning in Open-Vocabulary Emotion Recognition (NeurIPS 2026 · similarity 0.91)
- Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.89)
- Multi-to-Single: Reducing Multimodal Dependency in Emotion Recognition Through Contrastive Learning (AAAI 2025 · similarity 0.88)
- EMOE: Modality-Specific Enhanced Dynamic Emotion Experts (CVPR 2025 · similarity 0.88)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.