Research map / Vision-language and multimodal / Multimodal LLMs
Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Point-RFT trains Qwen2.5-VL-7B with SFT on a 71K Point-CoT dataset (GPT-4o rationales grounded with Molmo-7B points) followed by GRPO on ChartQA, yielding visually grounded chain-of-thought for chart and document reasoning.
Problem
Text-only chain-of-thought for vision-language reasoning suffers from visual hallucination and weak multimodal integration, and RL with outcome rewards is underexplored for multimodal reasoning.
Approach
Interleave step-by-step reasoning with explicit point coordinates grounding each referenced visual element, format-finetune on such data, then apply GRPO with format and accuracy rewards so the model learns to use grounded CoT.
Key results
- On ChartQA, Point-RFT reaches 90.04% accuracy versus 70.88% for the base Qwen2.5-VL-7B with CoT and 83.92% for text-only Base-RFT.
- Point-RFT averages 53.16% across five out-of-domain benchmarks, versus 33.80% for Base-RFT and 46.78% for Point-SFT.
- Ablation: 0-based XML pointing indexing gives 56.72% SFT accuracy versus 31.32% with random indexing; removing pointing drops SFT accuracy from 56.72% to 27.60%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add visually grounded Point-CoT format finetuning and GRPO instead of text-only CoT GRPO | Qwen2.5-VL-7B · ChartQA test | accuracy: 83.92 -> 90.04 | Helped |
| Add GRPO to Point-SFT | Qwen2.5-VL-7B · ChartQA test | accuracy: 87.21 -> 90.04 | Helped |
| Add visually grounded Point-CoT format finetuning and GRPO instead of text-only CoT GRPO | Qwen2.5-VL-7B · CharXiv validation | accuracy: 25.20 -> 36.20 | Helped |
| Add visually grounded Point-CoT format finetuning and GRPO instead of text-only CoT GRPO | Qwen2.5-VL-7B · PlotQA test | accuracy: 14.85 -> 20.40 | Helped |
| Add visually grounded Point-CoT format finetuning and GRPO instead of text-only CoT GRPO | Qwen2.5-VL-7B · IconQA test | accuracy: 48.55 -> 59.80 | Helped |
| Add visually grounded Point-CoT format finetuning and GRPO instead of text-only CoT GRPO | Qwen2.5-VL-7B · TabMWP mini-test | accuracy: 61.40 -> 70.90 | Helped |
| Add visual point annotations to supervised format finetuning (ablation) | Point-CoT SFT, Qwen2.5-VL-3B · ChartQA | accuracy: 27.60 -> 56.72 | Helped |
| Add visual point annotations to SFT and RL (ablation) | Point-RFT, Qwen2.5-VL-3B · ChartQA | accuracy: 77.16 -> 86.24 | Helped |
| Use XML rather than JSON for point annotations (ablation) | Point-SFT, Qwen2.5-VL-3B · ChartQA | accuracy: 49.36 -> 56.72 | Helped |
| Use standard point indexing rather than random indexing (ablation) | Point-SFT, Qwen2.5-VL-3B · ChartQA | accuracy: 31.32 -> 56.72 | Helped |
| Train Point-SFT for 500 rather than 300 steps (ablation) | Point-SFT, Qwen2.5-VL-3B · ChartQA | accuracy: 50.00 -> 56.72 | Helped |
| Train Point-SFT beyond 500 steps (ablation) | Point-SFT, Qwen2.5-VL-3B · ChartQA | accuracy: 56.72 -> 45.56; 56.72 -> 37.00; 56.72 -> 20.44 | Hurt |
Surprising findings
- Qwen2.5-VL-7B without CoT (87.30% on ChartQA) outperforms its CoT variant (70.88%), which the authors attribute to short-answer post-training.
- Using 0-based versus 1-based point indexing changes SFT accuracy dramatically (56.72% vs 31.32%).
Limitations
- Two-stage pipeline requires substantial compute and yields long autoregressive outputs with high inference latency.
- Evaluation focuses on chart and document benchmarks; OOD gains are uneven (e.g., PlotQA and IconQA changes are small, Qwen2-VL IconQA drops).
Experimental setup
Datasets: Point-CoT (71K, built from Mammoth-VL: ChartQA, DVQA, PlotQA, CLEVR, TallyQA) ChartQA training split
Benchmarks: ChartQA CharXiv PlotQA IconQA TabMWP Counting (SuperCLEVR)
Metrics: overall accuracy format compliance inner accuracy
Baselines: Base-RFT Point-SFT Qwen2.5-VL-7B Qwen2-VL-7B Molmo-7B
Builds on: GRPO Qwen2.5-VL Easy-R1 Mammoth-VL
Closest papers by meaning
- Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models (NeurIPS 2025 · similarity 0.92)
- TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding (ICML 2026 · similarity 0.92)
- Uni-CoT: Towards Unified Chain-of-Thought Reasoning Across Text and Vision (ICLR 2026 · similarity 0.91)
- GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinking (CVPR 2026 · similarity 0.91)
- See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs (CVPR 2026 · similarity 0.91)
- AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.