Research map / Vision-language and multimodal / Multimodal LLMs

Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Decomposes the on-policy distillation loss for VLMs into language-prior and visual-grounding gradients and steers the update toward the visual component (VGS), with a language-preservation regularizer, showing gains over standard on-policy distillation on seven VLM reasoning benchmarks.

Problem

On-policy distillation for vision-language reasoning optimizes a monolithic objective that implicitly mixes language-prior matching and visual grounding, and the resulting update may underweight visual perception, the apparent bottleneck for small VLM students.

Approach

Decompose the reverse-KL on-policy loss into a language-prior term and a visual-information-gain term, then add a weighted visual term (gradient steering with norm normalization) plus a selective language-preservation KL on high-visual-dependency tokens.

Key results

• On Qwen3-VL-8B to 2B distillation, VGS reaches 46.10% average greedy accuracy vs 43.74% for standard on-policy distillation (+2.37).
• On Qwen3-VL-8B to 4B distillation, VGS reaches 58.12% average greedy accuracy vs 56.64% for standard (+1.56).
• Combining GRPO with VGS-regularized distillation yields 47.20% average greedy accuracy on the 2B student vs 45.41% for GRPO + standard KD (+1.79 over that baseline).

Surprising findings

• Steering toward the language prior instead of the visual term degrades accuracy below the standard baseline.
• On VisualPuzzles with the 4B student, VGS is slightly below standard distillation in greedy accuracy (-0.44).

Limitations

• Training step cost increases by about 1.375x due to an extra forward pass without image input.
• Relies on the teacher's visual input having a distinguishable influence; if the teacher ignores the image, VGS reduces to standard distillation.
• Gradient-norm normalization factor is fixed to a precomputed constant per model rather than computed each step.

Experimental setup

Datasets: Vision-SR1-47K

Benchmarks: MMMU-Pro-4 LogicVista MathVerse-VD MathVerse-VO VisualPuzzles MathVision VlmsAreBlind

Metrics: Acc@1 (greedy) Acc@16 (T=1.0)

Baselines: Standard on-policy distillation GRPO GRPO + Standard-KD Language Steering

Builds on: on-policy distillation GRPO MiniLLM reverse KL distillation Vision-SR1

Closest papers by meaning

• Visual-Advantage On-Policy Distillation for Vision-Language Models (NeurIPS 2026 · similarity 0.91)
• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.90)
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.90)
• Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning (NeurIPS 2025 · similarity 0.90)
• Artemis: Structured Visual Reasoning for Perception Policy Learning (ICML 2026 · similarity 0.90)
• SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs (ICML 2026 · similarity 0.90)