Research map / Vision-language and multimodal / Multimodal LLMs
Actial: Activate Spatial Reasoning Ability of Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Viewpoint Learning with the Viewpoint-100K dataset and a two-stage SFT (with hybrid cold-start CoT) then GRPO on SAT to activate spatial reasoning in Qwen2.5-VL-7B.
Problem
Whether MLLMs capture cross-view 3D consistency needed for spatial reasoning; current MLLMs rely on 2D cues and perform near chance on viewpoint questions.
Approach
Teach MLLMs viewpoint (horizontal translation and rotation) questions built from MVImgNet camera parameters via SFT with hybrid cold-start pseudo-CoT, then refine with GRPO on the SAT dataset to transfer to out-of-domain spatial tasks.
Key results
- Actial-7B reaches 87.6 average on CV-Bench vs 71.2 for Qwen2.5-VL-7B-Instruct and 99.2 on BLINK multi-view vs 53.3 baseline.
- On 3DSRBench, Actial-7B averages 47.7 vs 45.8 for the baseline, still slightly behind GPT-4o-level or Qwen VLMax results.
- Qwen base model gets 12.9% on Viewpoint-100K test; SFT raises it to 92.2%, but GRPO after SFT drops it to 81.4%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply two-stage fine-tuning | Qwen2.5-VL-7B-Instruct · 3DSRBench | Avg.: 45.8 -> 47.7 | Helped |
| Apply two-stage fine-tuning | Qwen2.5-VL-7B-Instruct · CV-Bench | Avg.: 71.2 -> 83.5 | Helped |
| Apply two-stage fine-tuning | Qwen2.5-VL-7B-Instruct · BLINK spatial tasks | Avg.: 73.4 -> 87.6 | Helped |
| Apply two-stage fine-tuning | Qwen2.5-VL-7B-Instruct · MMSI-Bench | Avg.: 26.5 -> 28.9 | Helped |
| Add Viewpoint-100K SFT (ablation) | Qwen2.5-VL-7B-Instruct · Viewpoint-100K test | accuracy: 12.9% -> 92.2% | Helped |
| Add GRPO after SFT (ablation) | Qwen2.5-VL-7B-Instruct · Viewpoint-100K test | accuracy: 92.2% -> 81.4% | Hurt |
| Add knowledge injection (SFT) (ablation) | Actial-7B · 3DSRBench | Avg.: 47.6 -> 47.7 | No clear effect |
| Add knowledge injection (SFT) (ablation) | Actial-7B · CV-Bench | Avg.: 83.1 -> 83.5 | Helped |
| Add knowledge injection (SFT) (ablation) | Actial-7B · BLINK spatial tasks | Avg.: 74.9 -> 87.6 | Helped |
| Add generalization enhancement (GRPO) (ablation) | Actial-7B · 3DSRBench | Avg.: 46.6 -> 47.7 | Helped |
| Add generalization enhancement (GRPO) (ablation) | Actial-7B · CV-Bench | Avg.: 73.1 -> 83.5 | Helped |
| Add generalization enhancement (GRPO) (ablation) | Actial-7B · BLINK spatial tasks | Avg.: 86.6 -> 87.6 | Helped |
Surprising findings
- Applying GRPO after SFT lowered accuracy on Viewpoint-100K (92.2% to 81.4%).
- Human-assisted CoTs and direct GRPO on Viewpoint-100K caused a growing KL divergence, indicating strong 2D bias in pretrained models.
Limitations
- Dataset is object-centric and limited to horizontal translation and rotation in multiple-choice form.
- Model lags prior state of the art on 3DSRBench and generates overly long reasoning that truncates on MMSI-Bench MSR.
- GRPO phase can erode injected viewpoint knowledge when relying on outcome rewards.
Experimental setup
Datasets: Viewpoint-100K MVImgNet SAT pseudo CoTs generated by Gemini 2.5 Pro
Benchmarks: 3DSRBench CV-Bench BLINK MMSI-Bench Viewpoint-100K
Metrics: accuracy
Baselines: GPT-4o Gemini-1.5-Pro Qwen2.5-VL-7B-Instruct LLaVA-Vid-7B (+SAT) LLaVA-v1.5-13B (+SAT) Robopoint-13B
Builds on: GRPO Qwen2.5-VL SAT MVImgNet
Closest papers by meaning
- Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence (NeurIPS 2025 · similarity 0.92)
- Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs (NeurIPS 2025 · similarity 0.92)
- Eliciting Complex Spatial Reasoning in MLLMs through Wide-Baseline Matching (CVPR 2026 · similarity 0.92)
- Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs (AAAI 2026 · similarity 0.92)
- From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Models (ICML 2026 · similarity 0.92)
- S$^2$-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidance (CVPR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.