Research map / Vision-language and multimodal / Multimodal LLMs

VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VOLD, which transfers reasoning from a text-only Qwen3-8B teacher to Qwen2.5-VL-3B by SFT cold-start followed by GRPO combined with on-policy reverse-KL distillation, evaluated zero-shot on visual reasoning benchmarks.

Problem

Vision-language models lack abundant high-quality image-text reasoning data, while text-only reasoning resources are plentiful; how to transfer text-based reasoning to VLMs effectively is open.

Approach

Combine GRPO with on-policy reverse-KL distillation from a text-only teacher on the same student rollouts (replacing the reference-KL term), with reward-guided masking that distills only on incorrect rollouts, after an SFT cold-start that aligns student and teacher distributions.

Key results

• VOLD reaches 28.0% on MathVision vs 21.9% for the Qwen2.5-VL-3B base and 24.4% for X-Reasoner (replicated), using text-only training.
• VOLD achieves 32.0% on MMMU-Pro and 45.0% on LogicVista, versus 27.1% and 40.3% for the base model.
• Without SFT cold-start on teacher traces, on-policy distillation gives nearly no gain over SFT+RL (e.g., MMMU-Pro 31.0 vs 30.8).

Surprising findings

• SFT on unfiltered teacher traces degrades performance below the base model on several benchmarks before RL.
• Without teacher-aligned SFT, adding on-policy distillation yields no benefit, while the original DeepSeek-R1-generated MoT data does not benefit either.

Limitations

• SFT traces are not answer-verified, so incorrect teacher reasoning is included in the cold-start data.
• Gains depend on a sufficiently long SFT cold start; the approach is sensitive to alignment quality.
• Some baselines were trained on evaluation-set images, complicating fair comparison.

Experimental setup

Datasets: Mixture-of-Thoughts orz-57k Geo3K MMMU-Pro MMStar MathVision MathVista MathVerse WeMath DynaMath LogicVista

Benchmarks: MMMU-Pro MMStar MathVision MathVista MathVerse DynaMath WeMath LogicVista MME HallusionBench

Metrics: accuracy

Baselines: X-Reasoner VLAA-Thinker VLM-R1-Math Qwen2.5-VL-3B

Builds on: GRPO DAPO KDRL X-Reasoner on-policy distillation

Closest papers by meaning

• Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding (ICML 2026 · similarity 0.89)
• GFlowVLM: Enhancing Multi-step Reasoning in Vision-Language Models with Generative Flow Networks (CVPR 2025 · similarity 0.89)
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.89)
• Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.89)
• Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.89)
• Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (ICLR 2026 · similarity 0.88)