Research map / Vision-language and multimodal / Robot action and navigation
Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MIDAS, a two-stage pipeline (inverse dynamics pretraining on motion-following tasks, then multi-task finetuning) with a T5 prompt encoder plus residual connection and autoregressive per-dimension action decoding, reaching new SOTA on VIMA-BENCH and showing in-context learning on held-out tasks.
Problem
Training robots to follow multimodal prompts that interleave images and text, which requires understanding transition dynamics and fine-grained visual details, and dependencies among action dimensions.
Approach
Reformulate any robot trajectory as a motion-following task for inverse dynamics pretraining, augment a pretrained T5 prompt encoder with a residual connection from visual tokens, and decode each action dimension as an autoregressive token to model inter-dimension dependencies.
Key results
- With pretraining, MIDAS reaches 97.8% average success on VIMA-BENCH L1 versus 87.2% for VIMA.
- On L4 Task 10 (Follow Motion), success rises from 0% for VIMA to 41.0% with pretraining.
- In-context learning evaluation gives 36.2% overall success on held-out Twist/Follow Motion/Follow Order tasks, versus 23.3% without Modified FT and 0.7% without pretraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add inverse-dynamics pretraining (ablation) | MIDAS · VIMA-BENCH L1 | success rate (Avg): 91.6 -> 97.8 | Helped |
| add inverse-dynamics pretraining (ablation) | MIDAS · VIMA-BENCH L2 | success rate (Avg): 91.8 -> 97.9 | Helped |
| add inverse-dynamics pretraining (ablation) | MIDAS · VIMA-BENCH L3 | success rate (Avg): 88.3 -> 93.4 | Helped |
| add inverse-dynamics pretraining (ablation) | MIDAS · VIMA-BENCH L4 | success rate (Avg): 49.1 -> 59.1 | Helped |
| replace motion-following inverse-dynamics pretraining with masked-action pretraining (ablation) | MIDAS · VIMA-BENCH L1 | success rate (Overall): 97.8 -> 86.0 | Hurt |
| replace motion-following inverse-dynamics pretraining with masked-action pretraining (ablation) | MIDAS · VIMA-BENCH L2 | success rate (Overall): 97.9 -> 86.3 | Hurt |
| replace motion-following inverse-dynamics pretraining with masked-action pretraining (ablation) | MIDAS · VIMA-BENCH L3 | success rate (Overall): 93.4 -> 86.0 | Hurt |
| replace motion-following inverse-dynamics pretraining with masked-action pretraining (ablation) | MIDAS · VIMA-BENCH L4 | success rate (Overall): 59.1 -> 48.5 | Hurt |
| replace decoder-only architecture with encoder-decoder architecture (ablation) | MIDAS · VIMA-BENCH L1 | success rate (Overall): 97.8 -> 91.0 | Hurt |
| replace decoder-only architecture with encoder-decoder architecture (ablation) | MIDAS · VIMA-BENCH L2 | success rate (Overall): 97.9 -> 90.5 | Hurt |
| replace decoder-only architecture with encoder-decoder architecture (ablation) | MIDAS · VIMA-BENCH L3 | success rate (Overall): 93.4 -> 87.2 | Hurt |
| replace decoder-only architecture with encoder-decoder architecture (ablation) | MIDAS · VIMA-BENCH L4 | success rate (Overall): 59.1 -> 69.8 | Helped |
Surprising findings
- Masked (BERT-style) inverse dynamics pretraining does not help and can hurt multitask performance (86.0% vs 91.6% without pretraining on L1).
- Encoder-decoder variant scores higher on L4 Task 10 (85.5%) than the decoder-only MIDAS, though it is weaker on L1-L3.
Experimental setup
Datasets: VIMA-BENCH
Benchmarks: VIMA-BENCH
Metrics: success rate
Baselines: VIMA Gato OBJ Gato Flamingo GPT
Builds on: VIMA Frozen Gato SayCan
Closest papers by meaning
- Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation (CVPR 2025 · similarity 0.91)
- RoboMP$^2$: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models (ICML 2024 · similarity 0.91)
- ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation (CVPR 2024 · similarity 0.90)
- VLAS: Vision-Language-Action Model with Speech Instructions for Customized Robot Manipulation (ICLR 2025 · similarity 0.89)
- PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
- Vision-Language Foundation Models as Effective Robot Imitators (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.