Research map / Vision-language and multimodal / Robot action and navigation
Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Seer, an end-to-end Predictive Inverse Dynamics Model that jointly trains conditional visual foresight and inverse dynamics action prediction in a GPT-2-style Transformer, pre-trained on robot data and evaluated on LIBERO-LONG, CALVIN ABC-D, and real-world Franka tasks.
Problem
Scalable robotic manipulation policies: existing approaches either behavior-clone actions from large robot data or use vision pre-training/world models in decoupled two-stage pipelines, without closing the loop between vision prediction and action.
Approach
An end-to-end predictive inverse dynamics model: foresight ([FRS]) tokens predict future RGB latents, and action ([INV]) tokens attend to those foresight tokens via a unidirectional attention mask, so action prediction is conditioned on forecast vision and both losses train jointly during pre-training and fine-tuning.
Key results
- Seer achieves average length 4.28 on CALVIN ABC-D with Seer-Large, a new state of the art.
- On LIBERO-LONG, pre-training raises Seer's average success by about 9 points from 78.7%, and Seer uses 65M trainable parameters versus OpenVLA's 7B.
- Real-world average success rate rises from 60.0% (scratch) to 78.4% with DROID pre-training across four generalization tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DROID pre-training before fine-tuning | Seer · Real-world tasks (4-task average) | success rate: 60.0% to 78.4% | Helped |
| Add DROID pre-training before fine-tuning | Seer · Real-world tasks (4-task average) | Score: 32.8 to 39.5 | Helped |
| Add inverse dynamics prediction to fine-tuning alongside visual foresight (ablation) | Seer · CALVIN ABC-D | Avg. Len.: 3.41 -> 3.64 | Helped |
| Add visual foresight to fine-tuning behavior cloning (ablation) | Seer · CALVIN ABC-D | Avg. Len.: 3.31 -> 3.41 | Helped |
| Add inverse dynamics prediction to pre-training alongside visual foresight (ablation) | Seer · CALVIN ABC-D | Avg. Len.: 3.73 -> 3.98 | Helped |
| Add visual foresight to pre-training without inverse dynamics (ablation) | Seer · CALVIN ABC-D | Avg. Len.: 3.64 -> 3.73 | Helped |
| Pre-train on OXE instead of training from scratch | Seer · Real-world tasks (6-task average) | success rate: 53.3 -> 56.7 | Helped |
| Pre-train on OXE rather than DROID | Seer · Real-world tasks (6-task average) | success rate: 71.1 -> 56.7 | Hurt |
| Pre-train on DROID instead of OXE | Seer · Real-world tasks (6-task average) | Score: 26.8 -> 31.3 | Helped |
Surprising findings
- Pre-training on OXE gives only marginal gains and slightly hurts some high-precision real-world tasks.
- Visual pre-training alone (MVP, MPI) performs about as well as Seer without pre-training on LIBERO-LONG, suggesting vision-only pre-training is insufficient.
Limitations
- Only six real-world downstream tasks evaluated, with limited coverage of high-precision and contact-rich tasks.
- Cross-embodiment capability only tested in a limited OXE experiment, not broadly across robots.
Experimental setup
Datasets: DROID LIBERO-90 LIBERO-LONG CALVIN ABC-D Open X-Embodiment (OXE)
Benchmarks: LIBERO-LONG CALVIN ABC-D Real-world Franka tasks (six tasks)
Metrics: success rate score average task completion length (Avg. Len.)
Baselines: MTACT MVP MPI OpenVLA RoboFlamingo Susie CLOVER GR-1 3D Diffusor Actor
Builds on: MAE CLIP Perceiver Resampler GPT-2 Susie GR-1
Closest papers by meaning
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.92)
- VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation (NeurIPS 2024 · similarity 0.91)
- DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation (NeurIPS 2026 · similarity 0.90)
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations (ICML 2025 · similarity 0.90)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.90)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.