Research map / Vision-language and multimodal / Robot action and navigation
ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes LLaMA-Adapter MLLM with category, affordance and masked pose tasks plus chain-of-thought inference, impedance-based waypoint adaptation and test-time adaptation to predict end-effector contact poses for object-centric manipulation.
Problem
Learning-based robot manipulation trained on a few simulator categories generalizes poorly to many object categories; predicting contact points and end-effector directions needs interpretable, generalizable reasoning.
Approach
Transfer MLLM common-sense reasoning to object-centric manipulation by fine-tuning only injected adapters with a three-level curriculum (category identification, affordance prior reasoning via yes/no pixel questions, masked-language pose infilling), predict poses via chain-of-thought, and adapt at test time using real-world success/failure supervision on the V-Adapter.
Key results
- Ours achieves an average initial-movement success of 0.41 on train categories and 0.51 on test categories, versus 0.41 for Implicit3D on test categories... (Table 1 AVG: Ours 0.41 train-test split, Implicit3D 0.27 train / 0.41 test averages)
- Ablation: adding affordance prior reasoning gives +7% and MLM +5%, with full combination reaching 0.56 average success on train-category novel instances.
- VoxPoser success on opening drawer is 14.0% versus 69.0% for ManipLLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add object category identification to fine-tuning (ablation) | ManipLLM · Novel instances in train categories | success rate: 0.41 -> 0.44; +3% | Helped |
| Add masked language modeling to FT + OCI (ablation) | ManipLLM · Novel instances in train categories | success rate: 0.44 -> 0.49; +5% | Helped |
| Add affordance prior reasoning to FT + OCI + MLM (ablation) | ManipLLM · Novel instances in train categories | success rate: 0.49 -> 0.56; +7% | Helped |
| Add chain-of-thought reasoning at inference (ablation) | ManipLLM · Novel instances in train categories | success rate: 0.54 -> 0.56; +2% | Helped |
| Use active impedance adaptation instead of direct movement in a fixed direction (ablation) | ManipLLM · Long-distance movement | success rate: −3% | Helped |
| Use ManipLLM instead of VoxPoser | ManipLLM · Opening Drawer | success rate: 14.0% -> 69.0% | Helped |
Surprising findings
- Training only on a limited set of simulator categories and then using MLLM priors, the approach outperforms baselines on test categories without needing movable-part masks at test time.
Limitations
- Real-world evaluation limited to a small number of objects (about 5 trials per category) with a suction gripper.
- Relies on simulator-collected data and depth-based 3D projection, so sim-to-real gap remains and needs test-time adaptation.
Experimental setup
Datasets: PartNet-Mobility SAPIEN simulator-generated manipulation samples
Benchmarks: PartNet-Mobility (SAPIEN simulator, train/test categories) real-world household objects
Metrics: manipulation success rate long-distance movement success rate
Baselines: Where2Act UMPNet FlowBot3D Implicit3D VoxPoser
Builds on: LLaMA-Adapter LLaMA CLIP Flowbot3D Where2Act
Closest papers by meaning
- OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints (CVPR 2025 · similarity 0.91)
- RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models (ICML 2024 · similarity 0.90)
- Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation (CVPR 2025 · similarity 0.90)
- Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning (ICML 2024 · similarity 0.90)
- PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies? (NeurIPS 2025 · similarity 0.90)
- DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.