Research map / Vision-language and multimodal / Robot action and navigation
RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoboMP^2, a robotic perception-planning framework pairing an MLLM-based goal-conditioned perceptor (GCMP, EVA-CLIP + flan-T5-xl with LoRA) with a retrieval-augmented GPT-4V planner (RAMP) that picks relevant code policies as in-context demos, reporting ~10% gains on VIMABench and real-world tasks.
Problem
Embodied robotic manipulation agents generalize poorly to unseen tasks and scenes: end-to-end policies overfit to limited closed-loop data, prompt-based planners use fixed human-selected examples and ignore multimodal environment information, and existing perceptors (YOLO, CLIP, SAM+CLIP) cannot res
Approach
Combine a tailored MLLM perceptor that grounds complex referring expressions to coordinates with a planner that retrieves the k most relevant code policies via coarse TF-IDF retrieval and fine SentenceBERT reranking over a rewritten instruction, then feeds them with the scene image to GPT-4V to generate code plans.
Key results
- On VIMABench, RoboMP^2 averages 82.4% on L1-L4 versus VIMA's 81.6% on L1-L3 average (per the paper's comparison, around 10% gain on unseen L4 tasks: 68.0 vs 48.9 for VIMA).
- Ablation: removing the coarse retriever drops average success from 82.4 to 14.5; removing the whole RAMP drops it to 8.7.
- On the perception test set, GCMP reaches 99.0 mAP@0.3 on complex perception while SAM+CLIP and Shikra score 0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove the coarse retriever (ablation) | RoboMP² · VIMABench | average success rate: 82.4 -> 14.5 | Hurt |
| remove instruction rewriting (ablation) | RoboMP² · VIMABench | average success rate: 82.4 -> 77.4 | Hurt |
| remove semantic reorder (ablation) | RoboMP² · VIMABench | average success rate: 82.4 -> 77.5 | Hurt |
| remove retrieval and randomly select prompt programs (ablation) | RoboMP² · VIMABench | average success rate: 82.4 -> 8.7 | Hurt |
| set the coarse-retriever weight λ to 0 (ablation) | RoboMP² with TF-IDF · VIMABench | average success rate: 82.4 -> 74.1 | Hurt |
| set the coarse-retriever weight λ to 0.5 (ablation) | RoboMP² with TF-IDF · VIMABench | average success rate: 82.4 -> 72.4 | Hurt |
| set the coarse-retriever weight λ to 0.75 (ablation) | RoboMP² with TF-IDF · VIMABench | average success rate: 82.4 -> 72.6 | Hurt |
| set the coarse-retriever weight λ to 1 (ablation) | RoboMP² with TF-IDF · VIMABench | average success rate: 82.4 -> 73.9 | Hurt |
| replace TF-IDF with BM25 at λ=0.25 (ablation) | RoboMP² · VIMABench | average success rate: 82.4 -> 43.1 | Hurt |
| replace TF-IDF with SentenceBERT at λ=0.25 (ablation) | RoboMP² · VIMABench | average success rate: 82.4 -> 33.9 | Hurt |
| use multimodal GPT4V instead of text-only GPT4 (ablation) | RoboMP² planner · VIMABench | average success rate: 77.6 -> 82.4 | Helped |
| use multimodal GPT4V instead of text-only GPT3.5 (ablation) | RoboMP² planner · VIMABench | average success rate: 66.2 -> 82.4 | Helped |
Surprising findings
- Text-only GPT-4 planner achieves 77.6 average versus 82.4 for GPT-4V, showing a moderate benefit from visual input.
- Removing the coarse retriever collapses performance to 14.5, far worse than dropping only the reranker, indicating the retrieval step is critical.
Limitations
- Relies on closed GPT-4V/GPT-4 APIs for planning, limiting reproducibility and cost.
- Codebase of only about 50 programs from 10 sources restricts retrieval coverage.
- Real-world evaluation covers only a small number of tasks (7 in one place, 6 in another).
Experimental setup
Datasets: VIMABench robotic referring-expression instruction-tuning data (self-constructed)
Benchmarks: VIMABench enriched VIMABench test set real-world tasks perception test set
Metrics: success rate mAP@0.3 mAP@0.5 mAP@0.75
Baselines: Gato Flamingo VIMA RT-2 VisProg CaP I2A Shikra GPT-3.5 GPT-4
Builds on: VIMA Code as Policies VisProg BLIP-2 Shikra
Closest papers by meaning
- Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning (ICML 2024 · similarity 0.91)
- ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation (CVPR 2024 · similarity 0.90)
- Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics (NeurIPS 2025 · similarity 0.90)
- RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (NeurIPS 2025 · similarity 0.90)
- Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes (ICLR 2026 · similarity 0.90)
- RoboCodeX: Multimodal Code Generation for Robotic Behavior Synthesis (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.