Research map / Video / Video segmentation and tracking
Predicting What Matters: Robust Generalist Robot Policy Learning via Future Semantic Mask
Yunfan Lou, Xiaowei Chi, Xiaojie Zhang, Zezhong Qian, Chengxuan Li, Rongyu Zhang, Yaoxu Lyu, Guoyu Song, Chuyao Fu, Haoxuan Xu, Pengwei Wang, Shanghang Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| predict future semantic masks (MWM-C1) instead of RGB (Cosmos w/ IDM) | IDM-based world model · LIBERO average | success rate: 0.675 -> 0.810 | Improved |
| predict future semantic masks (MWM-C1) instead of RGB (Cosmos w/ IDM) | IDM-based world model · LIBERO-10 | success rate: 0.488 -> 0.704 | Improved |
| predict mask latent features (MWM-C2) instead of RGB (Cosmos w/ Latent IDM) | latent IDM world model · LIBERO average | success rate: 0.873 -> 0.918 | Improved |
| use predictive latent features with an action head instead of explicit decoded masks plus IDM (ablation) | MWM · LIBERO average | success rate: 0.810 -> 0.918 | Improved |
| add wrist-view input to the mask world model (ablation) | MWM-C2 · LIBERO average | success rate: 0.918 -> 0.983 | Improved |
| replace GE-ACT RGB world model with MWM | GE-ACT · LIBERO average | success rate: 0.965 -> 0.983 | Improved |
| replace GE-ACT RGB world model with MWM | GE-ACT · RLBench (6 tasks) | success rate: 30.8% -> 68.3% | Improved |
| replace FiS-VLA with MWM | FiS-VLA · RLBench (6 tasks) | success rate: 50.0% -> 68.3% | Improved |
| replace GE-ACT with MWM | GE-ACT · Real-world Franka | success rate: 23.8% -> 67.5% | Improved |
| replace pi0 with MWM | pi0 · Real-world Franka | success rate: 38.8% -> 67.5% | Improved |
| replace GE-ACT with MWM under appearance shifts | GE-ACT · Real-world OOD | OOD-SR: 12.5% -> 42.1% | Improved |
| replace GE-ACT with MWM under background shift | GE-ACT · Real-world background shift | success rate: 3.8% -> 27.5% | Improved |
Closest papers by meaning
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations (ICML 2025 · similarity 0.89)
- Prediction with Action: Visual Policy Learning via Joint Denoising Process (NeurIPS 2024 · similarity 0.89)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.88)
- Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training (NeurIPS 2024 · similarity 0.88)
- EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields (NeurIPS 2026 · similarity 0.88)
- Action Images: End-to-End Policy Learning via Multiview Video Generation (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
