Research map / Vision-language and multimodal / Robot action and navigation
Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CrayonRobo, a VLA-based manipulator that takes key-frame images overlaid with crayon-style 2D visual prompts (contact point, z/y axes, moving direction) plus text coordinates and predicts SE(3) contact poses and post-contact movement directions.
Problem
Task goals conveyed by language are ambiguous, while goal images and videos are overly detailed; robots need a goal specification that is both precise and non-redundant for manipulation, including low-level end-effector actions and high-level planning.
Approach
Overlay color-coded 2D visual prompts on key-frame RGB images (contact point, gripper z/y axes, next moving direction), mirror them in text coordinates, and train a LLaMA-Adapter-based VLA with text supervision, orthogonality, and 2D projection losses to predict 3D contact poses and movement directions; long-horizon tasks are executed by chaining key-frame prompts.
Key results
- Ours(s) with manual prompts reaches 0.79 average success across seen and unseen tasks, versus 0.47 for ManipLLM and 0.39 for Implicit3D (Table 1).
- Ablation: adding all four prompt types raises success from 0.42/0.37 (position only) to 0.74/0.72 (seen/unseen).
- Real-world zero-shot success rates range from 5/10 to 9/10 on single-step tasks and 3/5 to 8/10 on multi-step tasks with drawn prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the end-effector z-axis prompt to the contact-point prompt (ablation) | CrayonRobo · Seen tasks | success rate: 0.42 -> 0.55 | Helped |
| add the end-effector y-axis prompt (ablation) | CrayonRobo · Seen tasks | success rate: 0.55 -> 0.70 | Helped |
| add the post-contact movement-direction prompt (ablation) | CrayonRobo · Seen tasks | success rate: 0.70 -> 0.74 | Helped |
| add the end-effector z-axis prompt to the contact-point prompt (ablation) | CrayonRobo · Unseen tasks | success rate: 0.37 -> 0.50 | Helped |
| add the end-effector y-axis prompt (ablation) | CrayonRobo · Unseen tasks | success rate: 0.50 -> 0.68 | Helped |
| add the post-contact movement-direction prompt (ablation) | CrayonRobo · Unseen tasks | success rate: 0.68 -> 0.72 | Helped |
| use both visual and language prompts instead of language prompts alone (ablation) | CrayonRobo · Seen tasks | success rate: 0.69 -> 0.74 | Helped |
| use both visual and language prompts instead of language prompts alone (ablation) | CrayonRobo · Unseen tasks | success rate: 0.68 -> 0.72 | Helped |
| add 10% or 20% directional-prompt noise (ablation) | CrayonRobo · Seen and unseen tasks | success rate | No clear effect |
| add 30% or 40% directional-prompt noise (ablation) | CrayonRobo · Seen and unseen tasks | success rate | Hurt |
| use manual rather than automatic prompts | CrayonRobo · Heat toaster | real-world success: 2/5 -> 3/5 | Helped |
Surprising findings
- Fine-tuning without any prompts on a specific task's successful executions reached 5/10 and 6/10 on two multi-step tasks, suggesting prompt-free execution is possible after prompted rollouts.
- Prompt direction noise up to 20% produced performance comparable to noise-free inputs.
Limitations
- Cannot directly avoid obstacles; relies on an external motion planner such as cuRobo.
- Performance degrades when direction prompt noise reaches 30-40%.
- Relies on Grounded-DINO and GPT-4 for automatic prompt generation, with noisy contact points from bounding box centers.
Experimental setup
Datasets: PartNet-Mobility SAPIEN simulator interaction data
Benchmarks: SAPIEN simulator seen/unseen articulated object tasks real-world Franka manipulation tasks
Metrics: manipulation success rate
Baselines: Flowbot3D ManipLLM Implicit3D RT-Trajectory AVDC
Builds on: ManipLLM LLaMA-Adapter RT-Trajectory RT-Sketch
Closest papers by meaning
- Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning (ICML 2024 · similarity 0.91)
- OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (CVPR 2025 · similarity 0.91)
- Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization (NeurIPS 2025 · similarity 0.91)
- ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation (CVPR 2024 · similarity 0.90)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.90)
- VideoVLA: Video Generators Can Be Generalizable Robot Manipulators (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.