Research map / Vision-language and multimodal / Multimodal LLMs

Visual-Advantage On-Policy Distillation for Vision-Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VA-OPD, which uses a teacher-computed visual advantage (log-prob drop when fine-grained image detail is destroyed) to reweight rollouts and split token-level KL into high- and low-VA groups for on-policy distillation of VLMs.

Problem

Standard on-policy distillation of vision-language models improves output matching but does not strengthen the student's reliance on fine-grained visual input, since the few vision-critical tokens are diluted by uniform token-level KL averaging.

Approach

Defines visual advantage (VA) as the teacher's token-level log-prob difference with vs. without fine-grained visual detail, then uses it at two granularities: softmax rollout reweighting by trajectory-mean VA and separately normalized KL over high-VA and low-VA token groups.

Key results

• On Qwen3-VL-8B to 2B on Geometry3K, VA-OPD reaches Math Avg 48.3 vs 45.4 for Standard OPD (+2.9) and Visual Avg 66.1 vs 64.6 (+1.5).
• With ViRL39K training, VA-OPD's gain over Standard OPD grows to +3.8 Math Avg and +2.5 Visual Avg.
• Gains over Standard OPD increase with teacher size: +2.1 Math Avg at 4B, +2.9 at 8B, +3.7 at 32B (student fixed at 2B).

Surprising findings

• Standard OPD improves task accuracy while mean visual advantage on all tokens barely rises (0.07 to 0.10), whereas VA-OPD raises it to 0.16.
• Masking KL on the top-10% highest-VA tokens hurts accuracy, while masking an equal fraction of random or low-VA tokens has little effect.

Limitations

• VA is a teacher-relative counterfactual proxy, not a direct measure of internal perception such as attention or grounding.
• Gains on benchmarks solvable from text alone (e.g., MMStar, MMMU) are modest.
• Evaluated only with a 2B student and Qwen3-VL teachers; the paper does not test other model families.

Experimental setup

Datasets: Geometry3K ViRL39K

Benchmarks: WeMath MathVista MathVerse HallusionBench AI2D MMMU MMStar OCRBench

Metrics: avg@8 accuracy

Baselines: Base CoT-SFT Off-policy KD Standard OPD GRPO PAPO

Builds on: on-policy distillation MiniLLM GKD GRPO PAPO

Closest papers by meaning

• Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding (ICML 2026 · similarity 0.91)
• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.90)
• In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.89)
• Masking Teacher and Reinforcing Student for Distilling Vision-Language Models (CVPR 2026 · similarity 0.89)
• Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.89)
• Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation (NeurIPS 2026 · similarity 0.89)