Research map / Vision-language and multimodal / Robot action and navigation
OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OWMM-Agent, a VLM-based agent that tracks robot state and multi-view scene frames and emits function-call actions for open-world mobile manipulation, with a fine-tuned OWMM-VLM (8B/38B) trained on simulation-synthesized agentic data.
Problem
Open-world mobile manipulation requires instruction-conditioned global scene understanding, robot state tracking, and grounded action generation, which general VLMs handle poorly due to domain shift (rare grounding tasks, state tracking, embodiment priors).
Approach
A unified multi-image, multi-turn VLM agent outputs chain-of-thought, a textual robot-state summary, and bounding-box/coordinate actions that are passed to classical navigation and motion planners; the VLM is adapted via instruction fine-tuning on an agentic data pipeline that uses PDDL task sequences and simulator ground truth in Habitat.
Key results
- OWMM-VLM-38B reaches 21.90% full-task success in simulated episodes (51.52% with lenient distance threshold), versus 0.33% for GPT-4o+PIVOT and GPT-4o+RoboPoint.
- OWMM-VLM-38B achieves 97.85% ego-centric decision-making accuracy and 87.54% image retrieval, against 48.53% and 46.46% for GPT-4o.
- In real-world single-step evaluation on a Fetch robot (10 samples), OWMM-VLM-38B reaches 90% total accuracy versus 46.67% for GPT-4o-based baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune InternVL-2.5 with the OWMM agentic synthesis data | OWMM-VLM-38B · OWMM single-step | Ego-centric decision-making: 48.53\% -> 97.85\% | Helped |
| Fine-tune InternVL-2.5 with the OWMM agentic synthesis data | OWMM-VLM-38B · OWMM single-step | Image retrieval: 46.46\% -> 87.54\% | Helped |
| Increase training data from 0k to 15k samples | OWMM-VLM-8B · OWMM single-step | Ego-centric decision-making: 17.52\% -> 73.27\% | Helped |
| Increase training data from 0k to 15k samples | OWMM-VLM-8B · OWMM single-step | Image retrieval: 1.27\% -> 41.36\% | Helped |
| Vary scene and object diversity while holding the training set at 45k samples | OWMM-VLM-8B · OWMM single-step, 45k training samples | Ego-centric decision-making: 88.96\% to 91.56\% | No clear effect |
| Add beam search (ablation) | OWMM-VLM-8B · OWMM single-step | Ego-centric decision-making: 96.72\% -> 97.30\% | Helped |
| Add beam search (ablation) | OWMM-VLM-8B · OWMM single-step | Image retrieval: 79.04\% -> 78.47\% | No clear effect |
| Add beam search (ablation) | OWMM-VLM-8B · OWMM single-step | Object affordance grounding: 0.93(\pm0.14) -> 0.97(\pm0.14) | Helped |
| Replace bounding-box output with direct coordinates (ablation) | OWMM-VLM-8B · OWMM single-step | Object affordance grounding: 0.9251\rightarrow0.6542 | Hurt |
| Replace bounding-box output with direct coordinates (ablation) | OWMM-VLM-8B · OWMM single-step | Receptacle affordance grounding: 0.9060\rightarrow0.6479 | Hurt |
| Remove reasoning and summarization (ablation) | OWMM-VLM-8B · OWMM single-step | Image retrieval: 0.7904\rightarrow 0.6586 | Hurt |
| Remove reasoning and summarization (ablation) | OWMM-VLM-8B · OWMM single-step | Ego-centric decision-making: 0.9672\rightarrow0.9049 | Hurt |
Surprising findings
- Removing the reasoning and summarization components sharply reduces image retrieval (79.04% to 65.86%) and decision-making accuracy.
- Object and scene diversity had negligible effect on performance (fluctuations within about 5%), while data volume mattered more.
Limitations
- Relies on a pre-mapping phase with camera pose graph and 2D occupancy map.
- Fails to generalize across embodiments with different arm reach, and does not handle dexterous hands.
- Real-world evaluation is small (10 samples) and requires human confirmation before executing actions.
Experimental setup
Datasets: HSSD YCB Objects Google Scanned Objects
Benchmarks: OWMM simulated benchmark (Habitat/HomeRobot, Fetch)
Metrics: success rate image retrieval accuracy affordance grounding score dead loop count
Baselines: GPT-4o InternVL2.5-8B GPT-4o+PIVOT GPT-4o+RoboPoint RoboPoint PIVOT
Builds on: InternVL-2.5 ReKep HomeRobot PIVOT RoboPoint
Closest papers by meaning
- MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulation (CVPR 2025 · similarity 0.87)
- ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making (NeurIPS 2026 · similarity 0.86)
- AC-DiT: Adaptive Coordination Diffusion Transformer for Mobile Manipulation (NeurIPS 2025 · similarity 0.86)
- HAMSTER: Hierarchical Action Models for Open-World Robot Manipulation (ICLR 2025 · similarity 0.86)
- OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (CVPR 2025 · similarity 0.86)
- CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.