Research map / Vision-language and multimodal / Multimodal LLMs
AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MER-Caption/MER-Caption+ (model-led, human-assisted annotated emotion descriptions), the AffectGPT MLLM with a pre-fusion module on a Qwen2.5 LLM, and the MER-UniBench benchmark with tailored metrics.
Problem
MLLM-based emotion understanding lacks large-scale descriptive emotion annotations and a multimodal-centric framework; free-form MLLM outputs also lack suitable evaluation metrics.
Approach
A model-led, human-assisted annotation pipeline (SALMONN + Chat-UniVi + GPT-3.5 cue merging, filtered by TalkNet, description length, and model-based crowdsourcing classifiers) plus an MLLM that fuses audio and video with a Q-Former or attention pre-fusion module before the LLM.
Key results
- AffectGPT achieves a mean MER-UniBench score of 74.77, versus 64.17 for the next best MLLM (Emotion-LLaMA) in the audio+video+text setting.
- Replacing the MER-Caption+ training data with MER-Caption+ yields 74.77 vs 68.91 for MER-Caption and 58.16 for MAFW under the same architecture.
- In a user study over 20 overlapping samples, MER-Caption+ descriptions won 0.86 vs MERR-Coarse (0.04 losses) and 0.59 vs MERR-Fine (0.27 losses).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AffectGPT with audio, video, and text | MLLM emotion understanding · MER-UniBench mean | mean score: 64.17 -> 74.77 | Helped |
| Train on MER-Caption+ instead of MERR-Fine | AffectGPT · MER-UniBench mean | mean score: 64.55 -> 74.77 | Helped |
| Apply low-level and high-level emotion and sentiment filtering (ablation) | AffectGPT training data · MER-UniBench | mean score: 68.91 -> 74.77 | Helped |
| Add high-level emotion filtering after low-level filtering (ablation) | AffectGPT training data · MER-UniBench | mean score: 69.54 -> 73.78 | Helped |
| Add high-level sentiment filtering after low-level filtering (ablation) | AffectGPT training data · MER-UniBench | mean score: 69.54 -> 74.05 | Helped |
| Add high-level emotion and sentiment filtering (ablation) | AffectGPT training data · MER-UniBench | mean score: 69.54 -> 74.77 | Helped |
| Add pre-fusion operation (ablation) | AffectGPT · MER-UniBench | mean score: 72.95 -> 74.77 | Helped |
| Use Q-Former pre-fusion rather than no pre-fusion (ablation) | AffectGPT · MER-UniBench | mean score: 72.95 -> 74.16 | Helped |
| Use attention pre-fusion rather than no pre-fusion (ablation) | AffectGPT · MER-UniBench | mean score: 72.95 -> 74.77 | Helped |
| Use audio, face, and text instead of audio, frame, and text (ablation) | AffectGPT · MER-UniBench | mean score: 73.39 -> 74.77 | Helped |
| Add frames to audio, face, and text inputs (ablation) | AffectGPT · MER-UniBench | mean score: 74.77 -> 74.60 | No clear effect |
| Add LoRA fine-tuning to the LLM (ablation) | AffectGPT · MER-UniBench | mean score: 73.30 -> 74.77 | Helped |
Surprising findings
- Filtering general instruction datasets for emotion content did not help on LLaVA and VideoChat, suggesting detailed non-emotion descriptions carry useful emotional cues.
- Increasing input modality does not always improve performance; face-only input slightly outperforms frame input and combining them gives no gain.
Limitations
- MER-Caption+ descriptions are not manually verified and may contain inaccurate descriptions.
- Evaluation restricted to single-person videos; multi-person MER is left to future work.
- No metric yet for evaluating extra non-basic labels in basic emotion recognition.
Experimental setup
Datasets: MER-Caption MER-Caption+ MER2024 unlabeled portion MER2023 MER2024 MELD IEMOCAP CMU-MOSI CMU-MOSEI CH-SIMS CH-SIMS v2 OV-MERD+
Benchmarks: MER-UniBench MER2023 MER2024 MELD IEMOCAP CMU-MOSI CMU-MOSEI CH-SIMS CH-SIMS v2 OV-MERD+
Metrics: HIT set-based F-score (Fs) WAF ACC precision recall
Baselines: OneLLM SECap PandaGPT Qwen-Audio SALMONN Otter Video-LLaVA Video-ChatGPT VideoChat2 LLaMA-VID VideoChat Chat-UniVi
Builds on: Q-Former MERBench Chat-UniVi SALMONN
Closest papers by meaning
- OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition (ICML 2025 · similarity 0.92)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.92)
- AffectGPT-RL: Revealing Roles of Reinforcement Learning in Open-Vocabulary Emotion Recognition (NeurIPS 2026 · similarity 0.91)
- MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.91)
- Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models (ICML 2025 · similarity 0.91)
- Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.