Research map / Vision-language and multimodal / Multimodal LLMs

AVERE: Improving Audiovisual Emotion Reasoning with Preference Optimization

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces the EmoReAlM benchmark (4000 human-verified MCQA items) for audiovisual emotion reasoning and proposes AVEm-DPO, a DPO variant with prompt-based modality preferences, emotion-based response preferences, and text-prior debiasing that improves zero-shot emotion reasoning of audiovisual MLLMs.

Problem

Audiovisual MLLMs for emotion understanding make reasoning errors (spurious associations between emotions and irrelevant audiovisual cues) and perception errors (hallucinating cues driven by language priors), and existing benchmarks do not measure these well.

Approach

Combine DPO with prompt-based modality preferences (rejecting the input whose modality is irrelevant to the prompt), two rejected responses (video-relevant spurious and emotion-relevant hallucinated), and a text-prior penalty term that discounts responses explainable by the text-only LM backbone.

Key results

• On EmoReAlM stress test, AVEm-DPO raises the reference base model's audio stress F1 from 50.3 to 80.9 and visual from 59.9 to 94.6.
• AVEm-DPO improves DFEW UAR from 56.78 to 58.54 and RAVDESS UAR from 53.59 to 58.66 over the reference base model in zero-shot settings.
• In a 40-participant user study on EMER, AVEm-DPO model had 54.74% emotion-description preference versus at most 11.60% for baselines and the lowest inconsistency rate (4.67%).

Surprising findings

• Multimodal inputs can hurt emotion reasoning compared with single-modality inputs, per the paper's single-modality analysis.
• Naive DPO on audiovisual inputs sometimes slightly degrades performance relative to the base model (e.g., DFEW UAR 56.78 to 55.67).

Limitations

• Benchmark derived from DFEW may inherit its cultural biases and uses short videos (2-10 s).
• Poor performance on disgust recognition attributed to few training samples.
• Spurious audio cue-emotion associations remain only partially mitigated.

Experimental setup

Datasets: DFEW MAFW MER2025 EMER RAVDESS MER2023

Benchmarks: EmoReAlM DFEW RAVDESS MER2023 EMER

Metrics: accuracy precision recall F1 unweighted average recall weighted average recall GPT-based clue/label/spurious/hallucination scores user preference

Baselines: Naive-DPO Vista-DPO VideoLLaMA 2 OLA VITA-1.5 Qwen-2.5 Omni EmotionLLaMA MoSEAR

Builds on: DPO Vista-DPO EmotionLLaMA VCD

Closest papers by meaning

• EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models (CVPR 2026 · similarity 0.89)
• Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models (ICML 2025 · similarity 0.88)
• CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs (ICLR 2025 · similarity 0.88)
• EmotionThinker: Prosody-Aware Reinforcement Learning for Explainable Speech Emotion Reasoning (ICLR 2026 · similarity 0.88)
• EmoThinker: Advancing Visual-Acoustic Emotion Analysis via Structural Token Selection and Chain-of-Thought Reasoning (CVPR 2026 · similarity 0.88)
• Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach (ICLR 2026 · similarity 0.88)