Research map / Vision-language and multimodal / Multimodal LLMs

Omni-Perception Policy Optimization for Multimodal Emotion Reasoning

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes OPPO, an RL framework for emotion-oriented Omni-MLLMs that adds a cue-coverage reward and a KL loss restricted to modality-specific tokens under unimodal masking, plus a diagnostic MEP-Bench for utilization and faithfulness.

Problem

Emotion-oriented Omni-MLLMs underuse visual and acoustic cues in their reasoning and hallucinate modality-specific statements that are not supported by the actual input modality.

Approach

Reward reasoning clauses for semantically covering fine-grained ground-truth visual, audio, and emotion cues (embedding-based evidence routing), and apply a KL penalty under unimodal masking only to tokens matched to visual or acoustic cues, instead of to all tokens as in PAPO.

Key results

• OPPO reaches a mean of 81.05% on MER-UniBench, above the reproduced AffectGPT-R1 (75.60%) and the reported AffectGPT-R1 (79.98%).
• On MEP-Bench, OPPO raises evidence recall from 58.00% (Baseline) to 70.44% and video/audio masking accuracy from 59.20%/33.00% to 76.40%/50.60%.
• On MME-Emotion, OPPO improves the mean reasoning score from 62.7% to 68.1% over the Baseline.

Surprising findings

• Generic PAPO and PAPO-dual KL constraints on all tokens degrade utilization and faithfulness relative to the reward-only variant.
• Setting the reward weight alpha of 0.6 or higher causes format collapse in later training.

Limitations

• MEP-Bench is small (300 samples) and filtered to cue-rich instances.
• Reported results depend on hyperparameters (alpha, beta, rho, delta) and the reward is sensitive to the similarity threshold.
• Evaluation of AffectGPT and AffectGPT-R1 is reproduced under a single-checkpoint protocol that differs from the originally reported numbers.

Experimental setup

Datasets: MER-Caption+ OV-MERD+ MER23 MER24 MELD IEMOCAP MOSI MOSEI SIMS SIMSv2 MER2025-OV OV-MERD

Benchmarks: MER-UniBench MME-Emotion MEP-Bench

Metrics: emotion-wheel F1 hit rate weighted average F-score evidence recall POPE-style accuracy under masking (Acc@V, Acc@A) recognition/reasoning/CoT score

Baselines: AffectGPT AffectGPT-R1 HumanOmni-V2 PAPO PAPO-dual R1-Omni Emotion-LLaMA Qwen2.5-Omni

Builds on: GRPO PAPO AffectGPT-R1 MER-Caption+

Closest papers by meaning

• Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences (ICLR 2026 · similarity 0.88)
• A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.88)
• EMO-R3: Reflective Reinforcement Learning for Emotional Reasoning in Multimodal Large Language Models (CVPR 2026 · similarity 0.88)
• Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration (NeurIPS 2025 · similarity 0.88)
• Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs (NeurIPS 2026 · similarity 0.88)
• Unveiling the Cognitive Compass: Theory-of-Mind–Guided Multimodal Emotion Reasoning (ICLR 2026 · similarity 0.88)