Research map / Vision-language and multimodal / Multimodal LLMs

Perception-Aware Policy Optimization for Multimodal Reasoning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PAPO, a GRPO/DAPO-compatible policy gradient objective that maximizes a KL divergence between the policy's outputs given original vs. masked images (Implicit Perception Loss) plus a Double Entropy regularizer, improving multimodal RLVR across eight benchmarks.

Problem

RLVR for multimodal reasoning uses objectives designed for text-only settings, and 67% of errors in GRPO-trained LMMs are perception errors, so visually grounded responses are not incentivized.

Approach

Add an unbounded KL term between the policy's token distribution conditioned on the original image and on a patch-masked image, maximized so the response depends on visual content; stabilize with a double entropy loss on both distributions. It modifies only the optimization objective, not data, rollouts, or rewards.

Key results

• PAPO-G-3B improves average accuracy over GRPO-3B from 47.92 to 49.92 overall, and PAPO-D-7B improves over DAPO-7B by 17.54% relative overall gain (55.01 to 63.16 avg).
• On vision-dependent tasks, PAPO-D-7B gains 19.09% relative over DAPO-7B.
• PAPO reports a 30.5% reduction in perception errors on manually analyzed error cases.

Surprising findings

• Adding an unbounded perception KL term without regularization causes 'PAPO hacking' and model collapse, with larger models and higher gamma more prone.
• NoisyRollout gives inconsistent gains, decreasing performance on 4 of 9 benchmarks.

Limitations

• Not evaluated on larger model sizes or other model families such as InternVL.
• Training efficiency not optimized; perception KL applied uniformly to all tokens.
• Unbounded KL objective is prone to collapse (PAPO hacking) requiring entropy regularization.

Experimental setup

Datasets: ViRL39K

Benchmarks: Geometry3K MathVista We-Math MMK12 MathVerse LogicVista SuperClevr Counting MMMU-Pro MathVerse-V OCR-Bench-v2 MMLU-Pro

Metrics: avg@8 accuracy relative gain error category rate

Baselines: GRPO DAPO NoisyRollout GRPO + No KL_ref DAPO + entropy loss

Builds on: GRPO DAPO DeepSeek-R1 DINOv2

Closest papers by meaning

• Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward (ICLR 2026 · similarity 0.94)
• PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping (NeurIPS 2026 · similarity 0.93)
• Spotlight on Token Perception for Multimodal Reinforcement Learning (ICLR 2026 · similarity 0.93)
• Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning (CVPR 2026 · similarity 0.93)
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.92)
• CFPO: Counterfactual Policy Optimization for Multimodal Reasoning (ICML 2026 · similarity 0.92)