Research map / Video / Video generation
RoboScape: Physics-informed Embodied World Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoboScape, an autoregressive transformer embodied world model that jointly predicts RGB video and temporal depth and adds keypoint-dynamics token consistency, trained on ~6.5M AgiBotWorld clips.
Problem
Embodied world models generate visually plausible robot videos but often violate 3D geometry and motion/deformation physics, especially in contact-rich manipulation with deformable objects.
Approach
Integrates physics knowledge inside the world model via auxiliary joint training: a parallel temporal depth branch whose features are injected into the RGB branch, plus adaptive keypoint dynamics learning (top-K high-motion keypoints tracked by SpatialTracker, with token consistency loss and keypoint-guided attention), avoiding cascaded physics simulators or external models at inference.
Key results
- RoboScape achieves LPIPS 0.1259, PSNR 21.85, AbsRel 0.3600 and DeltaPSNR 3.34, outperforming IRASim, iVideoGPT, Genie and CogVideoX on all six metrics.
- Diffusion Policy on Robomimic Lift reaches 91% success with 200 synthetic clips versus 92% with real data.
- As a policy evaluator, RoboScape reaches Pearson correlation 0.953 with the ground-truth simulator.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RoboScape rather than the strongest baseline, Genie | Video generation · Video generation evaluation | PSNR: 19.7571 -> 21.8533 | Helped |
| Use RoboScape rather than the strongest baseline, Genie | Video generation · Video generation evaluation | ΔPSNR: 1.9871 -> 3.3435 | Helped |
| Add temporal depth learning (ablation) | RoboScape · Video generation evaluation | AbsRel: 0.3921 -> 0.3600 | Helped |
| Add temporal depth learning (ablation) | RoboScape · Video generation evaluation | PSNR: 21.9465 -> 21.8533 | Mixed |
| Add keypoint dynamics learning (ablation) | RoboScape · Video generation evaluation | ΔPSNR: 2.9462 -> 3.3435 | Helped |
| Add keypoint dynamics learning (ablation) | RoboScape · Video generation evaluation | PSNR: 21.7087 -> 21.8533 | Helped |
| Add both temporal depth and keypoint dynamics learning (ablation) | RoboScape · Video generation evaluation | ΔPSNR: 1.9871 -> 3.3435 | Helped |
| Add both temporal depth and keypoint dynamics learning (ablation) | RoboScape · Video generation evaluation | PSNR: 21.4873 -> 21.8533 | Helped |
| Increase RoboScape-generated training data from 50 to 200 examples | Diffusion Policy · Robomimic Lift | success rate: 40% -> 91% | Helped |
| Increase RoboScape-generated training data from 200 to 800 examples | π0 · LIBERO Average | success rate: 66.6% -> 79.1% | Helped |
Surprising findings
- Ablation shows removing depth learning slightly improves PSNR and DeltaPSNR (21.95 vs 21.85; 3.49 vs 3.34), and removing keypoint learning improves AbsRel and depth accuracy.
- Scaling data can slightly degrade geometric accuracy, attributed to overfitting to the last conditional frame in smaller datasets.
Limitations
- Policy evaluation relies on manual judgment of success in world-model rollouts.
- Geometric metrics may degrade with more data, and the approach was not tested on real robots (stated as future work).
Experimental setup
Datasets: AgiBotWorld-Beta Robomimic LIBERO
Benchmarks: AgiBotWorld-Beta clips Robomimic Lift LIBERO
Metrics: PSNR LPIPS AbsRel delta1 delta2 DeltaPSNR success rate Pearson correlation R^2
Baselines: IRASim iVideoGPT Genie CogVideoX
Builds on: MAGVIT-2 Video Depth Anything SpatialTracker Diffusion Policy
Closest papers by meaning
- EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation (NeurIPS 2025 · similarity 0.89)
- LongScape: Advancing Long-Horizon Embodied World Models with Context-Aware MoE (NeurIPS 2026 · similarity 0.89)
- WorldSimBench: Towards Video Generation Models as World Simulators (ICML 2025 · similarity 0.88)
- RoboAlign-R1: Distilled Multimodal Reward Alignment for Robot Video World Models (NeurIPS 2026 · similarity 0.87)
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.87)
- PhyMotion: Structured 3D Motion Reward for Physics-Grounded Human Video Generation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.