Research map / Vision-language and multimodal / Multimodal LLMs

PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PRPO, a token-level RLVR method that scores tokens by a KL-based Robust Visual Dependency metric and reshapes GRPO/DAPO advantages with a sigmoid-style gate, improving LVLM math and multimodal reasoning on Qwen2.5-VL-3B/7B.

Problem

Trajectory-level outcome rewards in RLVR give identical advantage to all tokens, so sparse visually grounded perceptual tokens get weak credit in multimodal reasoning.

Approach

Identify pivotal perceptual tokens via a dual-perturbation KL metric (strong masking for visual dependency, weak jitter for inconsistency, combined as RVD = S·ReLU(1 − S'/(S+η))) and multiply the sequence advantage per token by a bounded logistic gate of RVD (PAR).

Key results

• PRPO reaches 54.0 average at 3B (vs 30.7 base Qwen2.5-VL-3B and 52.1 for VPPO) per Table 1.
• PRPO reaches 62.5 average at 7B (vs 41.4 base and 60.8 for VPPO), best across the seven benchmarks.
• Ablation on Qwen2.5-VL-7B: removing PAR drops average to 60.2 and removing VI drops it to 61.2, both above DAPO at 50.6.

Surprising findings

• The reported +23.3% and +21.1% 'average gains' are absolute point differences over the untrained base (30.7→54.0, 41.4→62.5), which the paper labels as percentages.

Limitations

• Relies on two extra forward passes per token under perturbed images, adding rollout cost (not quantified in text).
• Gate hyperparameters (threshold percentile, M, m, k) tuned on the same benchmarks reported in the main table.
• Evaluated only on Qwen2.5-VL 3B and 7B with a 2048-token response cap.

Experimental setup

Datasets: ViRL39K

Benchmarks: Geo3k MathVista We-Math MMK12 MathVerse LogicVista MMMU-Pro MathVerse_V

Metrics: accuracy average accuracy

Baselines: GRPO DAPO PAPO-G PAPO-D VPPO MM-Eureka-7B VL-Rethinker-7B NoisyRollout-7B R1-ShareVL-7B

Builds on: GRPO DAPO VPPO PAPO

Closest papers by meaning

• Perception-Aware Policy Optimization for Multimodal Reasoning (ICLR 2026 · similarity 0.93)
• Spotlight on Token Perception for Multimodal Reinforcement Learning (ICLR 2026 · similarity 0.93)
• Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward (ICLR 2026 · similarity 0.90)
• CFPO: Counterfactual Policy Optimization for Multimodal Reasoning (ICML 2026 · similarity 0.90)
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
• Reinforcing Multimodal Reasoning Against Visual Degradation (NeurIPS 2026 · similarity 0.90)