Research map / Vision-language and multimodal / Robot action and navigation
RoboGround: Robotic Manipulation with Grounded Vision-Language Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ROBOGROUND, a robot manipulation policy that uses grounding masks from a fine-tuned GLaMM-style VLM as intermediate guidance (channel concatenation and grounded perceiver), plus a pipeline generating 24K demos with 112K diverse instructions on RoboCasa.
Problem
Robotic manipulation policies generalize poorly to novel objects, instructions, and cluttered scenes; existing intermediate representations are either coarse (language) or costly (goal images, point flows), and existing simulation datasets have limited instruction diversity and scene complexity.
Approach
Use pixel-level grounding masks for target object and placement area, produced by a VLM grounded on instructions, as an intermediate representation injected into a GR-1-style transformer policy both as an extra image channel and as attention guidance in a Grounded Perceiver; paired with an automated pipeline for diverse appearance, spatial, and commonsense instructions.
Key results
- On the pick-and-place appearance split, ROBOGROUND reaches 78.5/30.5 contact/success versus 49.5/13.8 for GR-1 in the simulated benchmark.
- Under unseen-class evaluation, the mask variant reaches 79.5/27.3 versus 53.0/17.5 without masks.
- Fine-tuning GLaMM on simulation plus VLM data raises grounding mIoU from 13.2 (zero-shot) to 48.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add grounding masks to the policy | RoboGround · RoboCasa Appearance pick-and-place | success rate: 13.8 -> 30.5 | Helped |
| add grounding masks to the policy | RoboGround · RoboCasa Spatial pick-and-place | success rate: 16.3 -> 33.5 | Helped |
| add grounding masks to the policy | RoboGround · RoboCasa Common-sense pick-and-place | success rate: 11.5 -> 30.0 | Helped |
| add grounding masks to the policy | RoboGround · RoboCasa Easy pick-and-place | success rate: 42.8 -> 43.3 | Helped |
| add channel concatenation and grounded perceiver (ablation) | RoboGround · RoboCasa Appearance pick-and-place (ablation subset) | success rate: 12 -> 32 | Helped |
| add channel concatenation (ablation) | RoboGround · RoboCasa Appearance pick-and-place (ablation subset) | success rate: 12 -> 26 | Helped |
| add grounded perceiver (ablation) | RoboGround · RoboCasa Appearance pick-and-place (ablation subset) | success rate: 12 -> 22 | Helped |
| replace point or bounding-box grounding with masks (ablation) | RoboGround · RoboCasa Appearance pick-and-place (ablation subset) | success rate: 12 -> 32 | Helped |
| fine-tune the grounded VLM on simulated data (ablation) | GLaMM · Simulated grounding data | mIoU: 13.2 -> 45.5 | Helped |
| add original VLM grounding data to simulated fine-tuning data (ablation) | GLaMM · Simulated grounding data | mIoU: 45.5 -> 48.2 | Helped |
| train with the generated diverse dataset rather than original data alone (ablation) | RoboGround · RoboCasa Appearance pick-and-place (ablation subset) | success rate: 6 -> 12 | Helped |
Surprising findings
- Ground-truth masks only modestly outperform predicted masks in ablations, suggesting the VLM's predicted masks are nearly as useful as oracle masks for the policy.
Limitations
- Grasp success lags well behind contact rate, indicating grasping remains a bottleneck.
- Masks are computed once at episode start, which fits only single-turn tasks where target and placement positions stay fixed.
- Evaluation is entirely in simulation (RoboCasa) without real-robot experiments.
Experimental setup
Datasets: RoboCasa Objaverse VLM grounding data
Benchmarks: RoboCasa
Metrics: contact rate success rate mIoU
Baselines: ACT BC-Transformer GR-1
Builds on: GLaMM GR-1 RoboCasa Perceiver
Closest papers by meaning
- RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulation (CVPR 2026 · similarity 0.89)
- OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (CVPR 2025 · similarity 0.89)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.89)
- Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization (NeurIPS 2025 · similarity 0.88)
- RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches (ICLR 2024 · similarity 0.88)
- Robotic Visual Instruction (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.