Research map / Vision-language and multimodal / Multimodal LLMs
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the MERR multimodal emotion dataset and Emotion-LLaMA, an MLLM that fuses HuBERT audio and MAE/VideoMAE/EVA visual features into a LLaMA2-chat model via LoRA instruction tuning, reporting state-of-the-art emotion recognition and reasoning results.
Problem
Existing MLLMs such as GPT-4V cannot process audio well and fail to recognize subtle facial micro-expressions, and emotion instruction-tuning data with multimodal descriptions is scarce.
Approach
Build a large auto-annotated multimodal emotion description dataset (MERR) using AU-based peak-frame selection and open tools (MiniGPT-v2, Qwen-Audio, LLaMA-3), then train an LLM with emotion-specific audio and multiview visual encoders projected into its token space, using coarse-to-fine instruction tuning.
Key results
- Achieves Clue Overlap 7.83 and Label Overlap 6.25 on EMER, the top scores among compared MLLMs.
- Reaches F1 0.9036 on MER2023-SEMI with audio, visual and text inputs, above VAT (0.8911).
- In zero-shot DFEW, obtains UAR 45.59 and WAR 59.37, above GPT-4V (47.69 UAR, 54.85 WAR) on WAR only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Emotion-LLaMA rather than GPT-4V in zero-shot DFEW evaluation | DFEW zero-shot emotion recognition · DFEW zero-shot | WAR | Helped |
| Use Emotion-LLaMA rather than GPT-4V in zero-shot DFEW evaluation | DFEW zero-shot emotion recognition · DFEW zero-shot | UAR | Hurt |
| Use Emotion-LLaMA rather than S2D after fine-tuning | DFEW emotion recognition · DFEW fine-tuning | UAR | Helped |
| Use Emotion-LLaMA rather than S2D after fine-tuning | DFEW emotion recognition · DFEW fine-tuning | WAR | Helped |
| Add text to the audio and video modalities (ablation) | Emotion-LLaMA · MER2023-SEMI | F1 | Helped |
| Use Emotion-LLaMA rather than VAT | MER2023-SEMI emotion recognition · MER2023-SEMI | F1 | Helped |
| Use Emotion-LLaMA rather than GPT-4V | MER2024-OV emotion recognition · MER2024-OV | Avg: 8.52% | Helped |
| Use coarse-grained MERR instructions instead of raw concatenated descriptions (ablation) | Emotion-LLaMA instruction tuning · EMER | Clue Overlap: 1.87 | Helped |
| Use coarse-grained MERR instructions instead of raw concatenated descriptions (ablation) | Emotion-LLaMA instruction tuning · EMER | Label Overlap: 1.25 | Helped |
| Use fine-grained MERR instructions instead of coarse-grained MERR instructions (ablation) | Emotion-LLaMA instruction tuning · EMER | Clue Overlap: 0.42 | Helped |
| Use fine-grained MERR instructions instead of coarse-grained MERR instructions (ablation) | Emotion-LLaMA instruction tuning · EMER | Label Overlap: 0.69 | Helped |
| Use multimodal instruction tuning rather than standard training (ablation) | Emotion-LLaMA · MER2023-SEMI | F1 | Helped |
Surprising findings
- Most MLLMs scored zero on the disgust category in DFEW, with GPT-4V reaching only 10.34%, which the authors attribute possibly to safety constraints on the term.
- Emotion-LLaMA scores 0.00 on disgust and 3.31 on fear in zero-shot DFEW, despite the highest overall WAR.
Limitations
- Coarse-grained MERR labels are auto-generated from AU-based pseudo-labels and LLaMA-3 refinement, with the neutral category rated lowest in human evaluation.
- MERR source videos are restricted by EULA, so only description JSON files are released.
- Evaluation of reasoning relies on ChatGPT scoring rather than human judgement.
Experimental setup
Datasets: MERR MER2023 DFEW MER2024 EMER
Benchmarks: EMER MER2023-SEMI MER2024-NOISE MER2024-OV DFEW
Metrics: Clue Overlap Label Overlap F1 UAR WAR average of accuracy and recall
Baselines: GPT-4V Video-LLaMA Video-ChatGPT PandaGPT VideoChat Valley AffectGPT S2D VAT
Builds on: Video-LLaMA MiniGPT-v2 LLaMA VideoMAE HuBERT
Closest papers by meaning
- AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models (ICML 2025 · similarity 0.90)
- Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models (ICML 2025 · similarity 0.90)
- MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models (ICLR 2026 · similarity 0.90)
- Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier (AAAI 2026 · similarity 0.90)
- OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition (ICML 2025 · similarity 0.89)
- Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.