Research map / Vision-language and multimodal / Multimodal LLMs
Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Reason-RFT, a two-stage pipeline of CoT SFT activation followed by GRPO reinforcement fine-tuning with rule-based rewards, evaluated on visual counting, structure perception, and spatial transformation with Qwen2-VL-2B/7B.
Problem
SFT with CoT annotations on visual reasoning tasks tends to overfit and generalizes poorly under domain shift, while pure RL lacks robustness; the paper seeks a training paradigm that generalizes better and is data-efficient.
Approach
Combine a small CoT SFT stage (1.6K samples) that activates reasoning with GRPO using verifiable format and accuracy rewards (including a tolerance-based numeric reward and a function-sequence reward), and evaluate on in-domain and domain-shifted splits.
Key results
- On the 2B model, Reason-RFT reaches an average of 61.77 on visual counting and 67.58 on spatial transformation versus 53.87 and 33.74 for Reason-RFT-Zero (2B).
- The 2B Reason-RFT surpasses GPT-4o by 31.7% on domain-shifted spatial transformation.
- Reason-RFT reaches about 70% of Reason-RFT-Zero's final performance using 3% of training data (1,600 samples) on 2B, and over 92% with 3% of data on 7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CoT-SFT reasoning activation before RL (ablation) | Reason-RFT vs Reason-RFT-Zero · Spatial Transformation | accuracy: 42.13 vs. 74.61 | Helped |
| Add CoT-SFT reasoning activation before RL (ablation) | Reason-RFT vs Reason-RFT-Zero · Visual Counting DS-M | accuracy: 5.20 vs. 28.40 | Helped |
| Apply GRPO reinforcement fine-tuning after CoT-SFT (ablation) | Qwen2-VL-7B-Instruct · Visual Counting DS-D and DS-M | accuracy: 12.8% higher | Helped |
| Apply GRPO reinforcement fine-tuning after CoT-SFT (ablation) | Qwen2-VL-2B-Instruct · Spatial Transformation DS-L and DS-R | accuracy: 21.04% | Helped |
| Add <summary> and <caption> visual tokens (ablation) | Reason-RFT-Zero · Spatial Transformation AVG | accuracy: 33.74 -> 38.01; 62.17 -> 64.27 | Helped |
| Add <summary> and <caption> visual tokens (ablation) | Reason-RFT · Spatial Transformation AVG | accuracy: 69.33 -> 65.99; 69.48 -> 69.09 | No clear effect |
| Use partial-credit reward (α=0.50, β=0.25) (ablation) | Reason-RFT · Spatial Transformation ID and DS | accuracy: 74.61 -> 79.18; 64.05 -> 56.36; 64.08 -> 55.45; 79.97 -> 80.89; 59.36 -> 53.20; 58.61 -> 52.61 | Mixed |
| Penalize partial matches (α=-0.25, β=-0.50) (ablation) | Reason-RFT · Spatial Transformation DS-L and DS-R | accuracy: 64.05 -> 64.41; 64.08 -> 64.72; 59.36 -> 64.83; 58.61 -> 63.18 | Helped |
| Replace GeoMath-only CoT activation data with mixed specific-domain data (ablation) | Reason-RFT · Structure Perception AVG | accuracy: 54.26 -> 48.02 | Hurt |
| Replace GeoMath-only CoT activation data with mixed general-domain data (ablation) | Reason-RFT · Structure Perception AVG | accuracy: 54.26 -> 41.38 | Hurt |
Surprising findings
- Partial-credit rewards improve in-domain accuracy but reduce domain-shift generalization, while penalizing partial matches helps generalization.
- Adding explicit caption/summary format tags helps Reason-RFT-Zero but has limited effect on Reason-RFT.
Limitations
- Longer reasoning traces in Reason-RFT from distilled CoT data may cause overthinking and computational overhead.
- Smaller models rely more heavily on CoT-SFT priors; mixing CoT data from broader domains reduces domain-specific performance.
- Evaluation limited to three synthetic/reconstructed visual reasoning tasks with Qwen2-VL backbones.
Experimental setup
Datasets: CLEVR-Math Super-CLEVR Geo170K Math360K Geometry3K TRANCE
Benchmarks: Super-CLEVR-Math (DS-D/DS-M) CLEVR-Math (ID) GeoMath-4K5 (ID) Geometry3K (DS) Trans-Center-6K (ID) Trans-Left-6K (DS-L) Trans-Right-6K (DS-R) MMMU RealWorldQA MathVision AI2D
Metrics: accuracy
Baselines: ANS-SFT CoT-SFT Reason-RFT-Zero GPT-4o Gemini-1.5-Pro Qwen2.5-VL-3B Qwen2.5-VL-7B InternVL-2.5-8B Phi-3.5-Vision-4B Llama-3.2-11B-Vision Pixtral-12B
Builds on: GRPO DeepSeek-R1 Open-R1 CLEVR-Math TRANCE
Closest papers by meaning
- More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.93)
- Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning (NeurIPS 2025 · similarity 0.93)
- Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning (NeurIPS 2025 · similarity 0.92)
- Activating Visual Context and Commonsense Reasoning Through Masked Prediction in VLMs (AAAI 2026 · similarity 0.92)
- RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning (ICLR 2026 · similarity 0.92)
- Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.