Research map / Vision-language and multimodal / Robot action and navigation
Robotic Visual Instruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Robotic Visual Instruction (RoVI), hand-drawn arrow/circle sketches on the observation to command 3D manipulation, with a VLM-plus-keypoint pipeline (VIEW) and a 15K-pair RoVI Book dataset for fine-tuning LLaVA.
Problem
Natural language lacks the spatial precision needed for robotic control (ambiguity, verbosity), while goal-image and trajectory inputs are hard for users to provide.
Approach
Users draw object-centric 2D symbols (arrows, circles, colors, numbers) on the initial observation; a VLM interprets them into language plans and code, a YOLOv8 keypoint module extracts arrow/circle keypoints, and a keypoint-conditioned low-level policy executes them in 3D.
Key results
- VIEW with GPT-4o or LLaVA-13B (RoVI Book) reaches 87.5% average real-world success over 8 tasks, vs. 43.8% for VoxPoser and 45% for CoPa in real world? (CoPa real-world average 30%).
- In simulation, VIEW reaches 82.5% (GPT-4o) and 87.5% average vs. 13.3% for Octo-goal-image and 3% for Octo-language.
- Fine-tuned YOLOv8 keypoint module achieves mAP 1.00 across four RoVI tasks, outperforming GDINO, OWL-ViT and OWL-V2 in mean distance and mAP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VIEW with RoVI-conditioned policies instead of language-conditioned CoPa or VoxPoser | Robotic manipulation · Real-world tasks 1–8 | success rate: 45 -> 87.5; 43.8 -> 87.5 | Helped |
| Use VIEW instead of RT-1-X or Octo policies | Robotic manipulation · Simulation tasks 1–3 | success rate: 20 -> 76.6; 13.3 -> 76.6; 3 -> 76.6 | Helped |
| Fine-tune LLaVA-13B on RoVI Book | LLaVA-13B · Real-world task 8 | task and planning success rate: 82 | Helped |
| Use Geometric rather than Loose drawing style (ablation) | RoVI instructions evaluated with VLMs · Modified Open X-Embodiment | task prediction and subgoal planning: 0.74 -> 0.8 | Helped |
| Use the trained YOLOv8 keypoint module rather than open-vocabulary detectors (ablation) | VIEW keypoint extraction · Keypoint extraction tasks 1–4 | mAP: 1.00 ± 0.00 | Helped |
Surprising findings
- Small VLMs (under 13B) without fine-tuning fail entirely at understanding RoVI, while GPT-4o, Gemini and Claude succeed via in-context learning.
- Geometric drawing style yields better VLM task comprehension than loose hand-drawn style.
Limitations
- Real-world evaluation limited to 8 real and 3 simulated tasks with 10 trials each.
- Large-model comprehension accuracy drops as the number of task steps increases.
- Small fine-tuned model only evaluated on RoVI comprehension; code and datasets promised but not yet released.
Experimental setup
Datasets: Open-X Embodiment RoVI Book
Benchmarks: SIMPLER real-world manipulation tasks (8 tasks)
Metrics: success rate spatiotemporal alignment (6-point Likert) task and planning accuracy mean distance mAP@50px
Baselines: VoxPoser CoPa RT-1-X Octo Claude 3.5-Sonnet Gemini-1.5 Pro
Builds on: VoxPoser CoPa Set-of-Mark AnyGrasp YOLOv8
Closest papers by meaning
- OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (CVPR 2025 · similarity 0.90)
- Localizing, Structuring, and Rendering: Bridging 3D and 2D Vision-Language-Action Models for Robotic Manipulation (CVPR 2026 · similarity 0.90)
- Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation (CVPR 2025 · similarity 0.89)
- TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies (ICLR 2025 · similarity 0.89)
- Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.89)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.