Research map / Vision-language and multimodal / Multimodal LLMs

Spotlight on Token Perception for Multimodal Reinforcement Learning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VPPO, a multimodal RLVR policy gradient that weights trajectory advantages by token-level visual dependency (KL between image and perturbed-image predictions) and applies gradient updates only to the top-40% most visually dependent tokens.

Problem

Multimodal RLVR for LVLMs treats all tokens and trajectories uniformly, ignoring which tokens and trajectories depend on visual perception, so correct but perception-free reasoning paths get the same learning signal as visually grounded ones.

Approach

Quantify each token's visual dependency as the KL divergence between the policy's next-token distribution given the real image versus a perturbed image; use trajectory-mean dependency to shape GRPO/DAPO advantages (TAS) and a top-k% token mask to filter policy gradients (TGF).

Key results

• VPPO on Qwen2.5-VL-7B reaches 57.5% average across eight benchmarks vs 55.0% for DAPO and 53.1% for GRPO (base 38.3%).
• VPPO on Qwen2.5-VL-32B reaches 64.6% average vs 63.5% for DAPO and 57.0% for the base model.
• Applying VPPO on top of GRPO raises Qwen2.5-VL-7B from 53.1% to 54.8%; ablation shows TGF alone gives 57.1% and TAS alone 56.3% on DAPO baseline.

Surprising findings

• Entropy-based token selection (high-entropy forking tokens) gives little gain in the multimodal setting, unlike text-only RLVR results.
• Without an entropy penalty, the DAPO baseline collapses into incoherent output, with average accuracy dropping to 49.0%.

Limitations

• Visual dependency requires an extra forward pass with perturbed images, adding computation.
• Gains over DAPO are modest (2.5 points at 7B, 1.1 at 32B) and based on single-run averages without reported variance.
• Relies on an entropy penalty of 0.06 for stability, applied to both baseline and VPPO.

Experimental setup

Datasets: ViRL39K MathVerse

Benchmarks: MathVerse DynaMath MMK12 Geo3k MathVision We-Math LogicVista MMMU-Pro A-OKVQA SimpleVQA

Metrics: avg@8 accuracy exact match

Baselines: GRPO DAPO MM-Eureka ThinkLite VL-Rethinker NoisyRollout R1-ShareVL PAPO-D

Builds on: GRPO DAPO EasyR1 ViRL39K

Closest papers by meaning

• Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward (ICLR 2026 · similarity 0.94)
• PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping (NeurIPS 2026 · similarity 0.93)
• Perception-Aware Policy Optimization for Multimodal Reasoning (ICLR 2026 · similarity 0.93)
• Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning (CVPR 2026 · similarity 0.91)
• Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning (ICML 2026 · similarity 0.91)
• Unleashing Perception-Time Scaling to Multimodal Reasoning Models (ICLR 2026 · similarity 0.90)