Research map / Vision-language and multimodal / Multimodal LLMs
OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OpenVLThinker-7B, trained from Qwen2.5-VL-7B by iterating lightweight SFT on R1-distilled reasoning traces with two-stage curriculum GRPO, reaching 47.9 average over six visual reasoning benchmarks with 12K RL samples per iteration.
Problem
Open-source large vision-language models rarely exhibit R1-style long chain-of-thought reasoning; naive distillation from text-only reasoning models via captions degrades visual performance, and pure RL struggles to surface reflective behaviors in 7B models.
Approach
Alternate SFT (distilling caption-grounded CoT from QwQ-32B, then self-generated CoT from the previous iteration) with two-stage source-based curriculum GRPO (medium then hard data), retraining from the base model each iteration; SFT is argued to surface reasoning "action" tokens such as "wait" that RL then refines.
Key results
- OpenVLThinker-7B averages 47.9 across six benchmarks vs 45.6 for Qwen2.5-VL-7B, with MathVista 72.3 and HallusionBench 69.1.
- Using 12K RL samples, OpenVLThinker-7B outperforms VLAA-Thinker-7B (150K data, avg 46.6) and R1-Onevision-7B (165K data, avg 41.4).
- OpenVLThinker-3B reaches 63.4 on MathVista, above Qwen2.5-VL-3B (62.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply iterative SFT→RL rather than RL-only (ablation) | Qwen2.5-VL-7B · HallusionBench | accuracy: 66.8 -> 70.2 | Helped |
| apply iterative SFT→RL rather than SFT-only (ablation) | Qwen2.5-VL-7B · EMMA | accuracy: 22.3 -> 25.2 | Helped |
| restart from the base model at each iteration rather than continue training (ablation) | OpenVLThinker · HallusionBench | accuracy: 66.8 -> 70.2 | Helped |
| use Qwen2.5-VL-7B rather than Qwen2.5-VL-3B to generate initial captions (ablation) | OpenVLThinker iterative training · MathVista | accuracy: 70.2 -> 71.7 | Helped |
| refine captions with one round of feedback rather than use original captions (ablation) | caption-based VQA reasoning · ConTextual validation | pass@1: 33 -> 29 | Hurt |
| refine captions with one round of feedback rather than use original captions (ablation) | caption-based VQA reasoning · ConTextual validation | pass@4: 44 -> 46 | Helped |
| progress from medium-difficulty RL data to hard data rather than train on hard data only (ablation) | GRPO-Iter1 · MathVision | accuracy | Helped |
| use medium-difficulty data instead of mixing easy and medium data (ablation) | GRPO-Iter1 · visual reasoning benchmarks | performance | Helped |
| distill initial SFT traces from image captions | Qwen2.5-VL-7B · MathVista | accuracy | Hurt |
| apply GRPO after SFT-Iter1 | Qwen2.5-VL-7B · MathVista | accuracy: +5.2% | Helped |
| use OpenVLThinker-7B rather than Qwen2.5-VL-7B | Qwen2.5-VL-7B · MathVista | accuracy: 68.5 -> 72.3 | Helped |
| use OpenVLThinker-7B rather than Qwen2.5-VL-7B | Qwen2.5-VL-7B · MathVision | accuracy: 24.0 -> 25.9 | Helped |
Surprising findings
- SFT-Iter1 initially causes performance drops, and direct RL from the base model without SFT reaches lower HallusionBench accuracy than the iterative approach.
- Continued training from the previous checkpoint degrades HallusionBench (66.8 vs 70.2 for re-training from base), suggesting overfitting.
Limitations
- Validated only at 7B (plus one 3B model distilled from 7B); scaling to larger backbones untested.
- Evaluation limited to a set of six (plus two additional) benchmarks; robustness in other tasks or real-world settings not probed.
- Substantial compute per iteration (about 20 hours RL per iteration on H100s).
Experimental setup
Datasets: LLaVA-OneVision MathV360K FigureQA MapQA GeoQA
Benchmarks: MathVista MathVerse MathVision MMMU-Pro EMMA HallusionBench MM-Star WeMath
Metrics: accuracy
Baselines: Qwen2.5-VL-7B GPT-4o Claude-3.5-Sonnet Mulberry-7B InternVL2.5-8B Kimi-VL-16B Qwen2-VL-7B R1-VL-7B R1-Onevision-7B VLAA-Thinker-7B
Builds on: GRPO DeepSeek-R1 ReST-EM iterative DPO DAPO Qwen2.5-VL
Closest papers by meaning
- Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning (NeurIPS 2025 · similarity 0.92)
- Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models (NeurIPS 2025 · similarity 0.91)
- HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning (NeurIPS 2026 · similarity 0.91)
- Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.91)
- Activating Visual Context and Commonsense Reasoning Through Masked Prediction in VLMs (AAAI 2026 · similarity 0.91)
- Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.