Research map / Vision-language and multimodal / Multimodal LLMs
RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoboRefer, a 3D-aware VLM for robotic spatial referring that adds a dedicated depth encoder via SFT and multi-step reasoning via GRPO-based RFT with metric-sensitive process rewards, trained on the new RefSpatial dataset and evaluated on RefSpatial-Bench.
Problem
Robots need to localize points referred to by spatially constrained, multi-step instructions in cluttered 3D scenes, but existing VLMs handle only single-step spatial understanding well and lack multi-step reasoning data and benchmarks.
Approach
Use a separate depth encoder and projector (initialized from the RGB ones) so depth does not interfere with the RGB encoder, then apply GRPO with outcome rewards plus metric-sensitive process rewards on key reasoning steps, trained on a simulation-backed multi-step dataset.
Key results
- RoboRefer-2B-SFT and 8B-SFT reach average spatial understanding success of 89.6% per abstract; 8B-SFT scores 96.90 on CV-Bench 2D-Relation.
- RoboRefer-2B-RFT surpasses Gemini-2.5-Pro by 17.4% absolute on RefSpatial-Bench (average).
- On simulated Open6DOR V2, the system with RoboRefer reaches 79.2% average success vs 72.4% for SoFar and reduces execution time to 29 s vs 40 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune NVILA-8B with RoboRefer SFT | NVILA-8B · CV-Bench 3D-Depth | accuracy: 91.83 -> 98.33 | Helped |
| Add RFT after SFT | RoboRefer-2B · RefSpatial-Bench Unseen | success rate: 33.77 -> 41.56 | Helped |
| Add 2D data to the SFT recipe (ablation) | RoboRefer SFT · BLINK_val | accuracy: 74.48 -> 89.27 | Helped |
| Add 3D data to the SFT recipe (ablation) | RoboRefer SFT · CV-Bench | accuracy: 81.83 -> 94.77 | Helped |
| Add simulated data to the SFT recipe (ablation) | RoboRefer SFT · CV-Bench | accuracy: 83.96 -> 94.77 | Helped |
| Use a dedicated rather than shared depth encoder (ablation) | RoboRefer SFT · BLINK_val | accuracy: 85.27 -> 89.27 | Helped |
| Add a depth encoder (ablation) | RoboRefer RFT · RefSpatial-Bench | success rate: 40.00 -> 53.00 | Helped |
| Add process reward to RFT (ablation) | RoboRefer RFT · RefSpatial-Bench | success rate: 48.00 -> 53.00 | Helped |
| Add RFT after SFT | RoboRefer-2B · RefSpatial-Bench Location | success rate: 47.00 -> 52.00 | Helped |
| Add RFT after SFT | RoboRefer-2B · RefSpatial-Bench Placement | success rate: 48.00 -> 54.00 | Helped |
| Use RoboRefer point predictions for localization and placement | Open6DOR V2 position track system · Open6DOR V2 position track | average success rate: 72.4 -> 79.2 | Helped |
| Use RoboRefer instead of GPT-4o for object localization and placement | Open6DOR V2 position track system · Open6DOR V2 position track | execution time: 40 -> 29 | Helped |
Surprising findings
- Qwen2.5-VL-7B finetuned on RefSpatial-style data also improves strongly, suggesting the dataset is model-agnostic.
- Gemini-2.5-Pro, despite strong 2D referring, struggles with distance-based 3D relations and multi-constraint referring.
Limitations
- Only the 2B model is trained with RFT due to compute limits.
- Predicts 2D image-plane points, requiring depth-based conversion to 3D.
- Struggles with brief or ambiguous human intent in instructions.
Experimental setup
Datasets: RefSpatial OpenImages CA-1M Infinigen Objaverse RoboRefIt Where2Place RoboSpatial CV-Bench BLINK SAT EmbSpatial
Benchmarks: CV-Bench BLINK RoboSpatial SAT EmbSpatial RefSpatial-Bench RoboRefIt Where2Place RefCOCO RefCOCO+ RefCOCOg MME
Metrics: accuracy success rate mask-contained point success rate execution time
Baselines: GPT-4o Gemini-2.5-Pro Claude-3.7-Sonnet NVILA-2B NVILA-8B Qwen-2.5-VL-7B Qwen-2.5-VL-72B SpatialBot-3B SpatialRGPT-8B SpaceLLaVA-13B RoboPoint-13B Molmo-7B
Builds on: NVILA GRPO RoboPoint SpatialRGPT Qwen2.5-VL
Closest papers by meaning
- RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics (CVPR 2025 · similarity 0.92)
- Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.90)
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- Spatial Understanding from Videos: Structured Prompts Meet Simulation Data (NeurIPS 2025 · similarity 0.90)
- Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics (NeurIPS 2025 · similarity 0.89)
- SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.