Research map / Vision-language and multimodal / Robot action and navigation
SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces semantic orientation (language-grounded object directions), the OrienText300K dataset, the PointSO predictor, and the SoFar VLM-agent system for 6-DoF spatial reasoning and manipulation.
Problem
VLMs and robot planners reason about object position but largely ignore object orientation, which is needed for 6-DoF fine-grained manipulation; existing orientation representations depend on reference frames or templates and lack semantic grounding.
Approach
Defines semantic orientation as a unit vector tied to a natural-language description (e.g., "handle", "plug-in" direction), learns it at scale from GPT-4o-annotated Objaverse objects with a point-cloud transformer (PointSO), and plugs it into a VLM scene graph with CoT reasoning to produce 6-DoF rearrangement plans.
Key results
- Zero-shot 48.7% overall success on Open6DOR 6-DoF track (perception setting), vs 35.6% for Open6DOR-GPT.
- Zero-shot 74.9% average success on SIMPLER (Visual Matching, Google Robot) per abstract; Variant Aggregation average 67.6%.
- 43.9% total on 6-DoF SpatialBench, versus 36.2% for GPT-4o.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add semantic orientation to the scene graph (ablation) | SoFar · Open6DOR V1 Rotation Track | success rate: 13.0 -> 57.0 | Helped |
| Add chain-of-thought reasoning (ablation) | SoFar with semantic orientation · Open6DOR V1 Position Track overall | success rate: 93.3 -> 93.0 | No clear effect |
| Add chain-of-thought reasoning (ablation) | SoFar with semantic orientation · Open6DOR V1 Rotation Track overall | success rate: 12.9 -> 57.0 | Helped |
| Increase training data from 15K to 350K (ablation) | PointSO-Base · OrienText300K validation split | average accuracy: 42.58 -> 72.46 | Helped |
| Use addition instead of cross-attention fusion (ablation) | PointSO-Base · OrienText300K validation split | average accuracy: 66.21 -> 72.46 | Helped |
| Use addition instead of multiplication fusion (ablation) | PointSO-Base · OrienText300K validation split | average accuracy: 65.04 -> 72.46 | Helped |
| Use addition instead of concatenation fusion (ablation) | PointSO-Base · OrienText300K validation split | average accuracy: 55.86 -> 72.46 | Helped |
| Replace Florence-2 with Grounding DINO (ablation) | SoFar · Open6DOR V1 6-DoF Track | success rate: 48.7 -> 44.6 | Hurt |
| Replace Florence-2 with YOLO-World (ablation) | SoFar · Open6DOR V1 6-DoF Track | success rate: 48.7 -> 27.8 | Hurt |
| Use SoFar rather than the strongest listed baseline | SIMPLER-Env WidowX + Bridge · SIMPLER-Env WidowX + Bridge | average success rate: 0.427 -> 0.583 | Helped |
| Use SoFar rather than GPT-4o | 6-DoF SpatialBench · 6-DoF SpatialBench | Total: 36.2 -> 43.9 | Helped |
Surprising findings
- Removing semantic orientation drops the Open6DOR 6-DoF overall from 48.7 to 14.2 with CoT alone; CoT by itself gives almost no gain on rotation tasks.
Limitations
- Decoupled pipeline: errors in grasping, perception, or motion planning propagate to execution.
- Orientation precision below 5 degrees is hard, and upright/upside-down tasks are a common failure mode.
- PointSO accuracy at 5 degrees is only about 60%.
Experimental setup
Datasets: Objaverse OrienText300K Open6DOR SIMPLER LIBERO EmbSpatial-Bench PartNet-Mobility
Benchmarks: Open6DOR Open6DOR V2 SIMPLER 6-DoF SpatialBench EmbSpatial-Bench PartNet-Mobility (SAPIEN) OrienText300K validation
Metrics: success rate angular accuracy at 45/30/15/5 degrees time cost
Baselines: GPT-4V Dream2Real VoxPoser Open6DOR-GPT Octo OpenVLA RT-2-X SpatialVLA
Builds on: Open6DOR CoPa ReKep Objaverse CLIP
Closest papers by meaning
- OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (CVPR 2025 · similarity 0.90)
- SpinBench: Perspective and Rotation as a Lens on Spatial Reasoning in VLMs (ICLR 2026 · similarity 0.90)
- ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulation (CVPR 2026 · similarity 0.89)
- OBJECT-UNI: A Unified Model for Object-Centric Spatial Understanding and Controllable Generation (NeurIPS 2026 · similarity 0.89)
- Localizing, Structuring, and Rendering: Bridging 3D and 2D Vision-Language-Action Models for Robotic Manipulation (CVPR 2026 · similarity 0.89)
- BOP-ASK: Object-Interaction Reasoning for Vision-Language Models (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.