Research map / Vision-language and multimodal / Multimodal LLMs
Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViLaSR, a Qwen2.5-VL-7B-based LVLM that reasons by interleaving text with self-issued bounding-box and auxiliary-line drawing operations, trained via cold-start SFT, reflective rejection sampling, and GRPO-based RL.
Problem
Text-centric multimodal reasoning in LVLMs struggles with spatial reasoning tasks requiring precise geometric understanding and continuous spatial tracking, and prior tool-integrated visual reasoning relies on fixed external perception tools and oversimplified reasoning data.
Approach
Equip the LVLM with elementary drawing operations (bounding-box annotation and auxiliary-line drawing) executed on the images so it reasons by editing the visual space itself, trained with a three-stage pipeline (synthetic cold start, reflective rejection sampling that filters for self-correction behavior, and RL with correctness plus format reward) rather than relying on external perception tools
Key results
- ViLaSR achieves 98.2 on MAZE versus 33.7 for Qwen2.5-VL-7B without reasoning, and 45.4 average on VSI-Bench, the best among the compared models.
- Reported average improvement of 18.4% over strong baselines across benchmarks.
- Removing reflective rejection sampling reduces reflection pattern exhibition by 96.5%, and removing it lowers several VSI-Bench subtask scores slightly.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with the three-stage drawing-and-thinking framework | Qwen2.5-VL-7B · MAZE | accuracy: +64.5 | Helped |
| train with the three-stage drawing-and-thinking framework | Qwen2.5-VL-7B · SpatialEval-Real | accuracy: +5.4 | Helped |
| train with the three-stage drawing-and-thinking framework | Qwen2.5-VL-7B · VSI-Bench | average score: +12.7 | Helped |
| train with the three-stage drawing-and-thinking framework | Qwen2.5-VL-7B · SPAR-Bench | accuracy: +5.9 | Helped |
| train with the three-stage drawing-and-thinking framework | Qwen2.5-VL-7B · MMSI-Bench | accuracy: +3.3 | Helped |
| add reflective rejection sampling (ablation) | ViLaSR · VSI-Bench | Absolute Distance score: +0.6% | Helped |
| add reflective rejection sampling (ablation) | ViLaSR · VSI-Bench | Room Size score: +0.7% | Helped |
| add reflective rejection sampling (ablation) | ViLaSR · VSI-Bench | Relative Distance score: +1.0% | Helped |
| add reflective rejection sampling (ablation) | ViLaSR · VSI-Bench | Object Count score: -0.1% | Hurt |
| add reflective rejection sampling (ablation) | ViLaSR · VSI-Bench | Relative Direction score: -0.9% | Hurt |
| add reinforcement learning (ablation) | ViLaSR · VSI-Bench numerical questions | average performance: -9.21% | Helped |
| add reinforcement learning (ablation) | ViLaSR · VSI-Bench multiple-choice questions | average performance: -4.07% | Helped |
Surprising findings
- Open-source Qwen2.5-VL-7B shows minimal or negative gains from enabling textual reasoning, while proprietary models gain substantially.
- The drawing-based framework without training performs worse than the Qwen2.5-VL-7B backbone, indicating prompting alone cannot elicit spatial drawing skills.
Limitations
- Training restricted to multiple-choice and numerical questions, excluding free-form spatial reasoning.
- Drawing operations limited to 2D, so complex 3D relationships and viewpoint changes are not well captured.
- Weaker on holistic tasks such as Room Size (-9.9 vs Qwen2.5-VL-7B baseline) and Route Plan (-2.1).
Experimental setup
Datasets: GQA VSR OpenImages OpenSpaces SpaceLLaVA GPT4Scene SR-91k procedurally generated mazes
Benchmarks: MAZE SpatialEval-Real VSI-Bench SPAR-Bench MMSI-Bench
Metrics: accuracy Mean Relative Accuracy (MRA)
Baselines: GPT-4o Gemini-1.5-Pro Gemini-2.0-Flash OpenAI o3 OpenAI o4-mini Qwen2.5-VL-7B Qwen2.5-VL-72B LLaVA-NeXT-Video-7B LLaVA-OneVision-72B CogCoM VisCoT SpaceR-7B
Builds on: GRPO Qwen2.5-VL VERL VisCoT CogCoM
Closest papers by meaning
- Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models (NeurIPS 2024 · similarity 0.93)
- Imagine While Reasoning in Space: Multimodal Visualization-of-Thought (ICML 2025 · similarity 0.92)
- Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulations (ICLR 2026 · similarity 0.92)
- Enhancing Spatial Reasoning Through Visual and Textual Thinking (AAAI 2026 · similarity 0.92)
- Visual Structures Help Visual Reasoning: Addressing the Binding Problem in LVLMs (NeurIPS 2025 · similarity 0.91)
- EgoMind: Activating Spatial Cognition through Linguistic Reasoning in MLLMs (CVPR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.