Research map / Vision-language and multimodal / Robot action and navigation
4D Visual Pre-training for Robot Learning
Chengkai Hou, Yanjie Ze, Yankai Fu, Zeyu Gao, Songbo Hu, Yue Yu, Shanghang Zhang, Huazhe Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use history frame instead of current frame as pre-training input (ablation) | DP3+FVP · UR5 real task PickSquare | success count (out of 20 trials): 15/20 -> 20/20 | Improved |
| use history frame instead of current frame as pre-training input (ablation) | DP3+FVP · UR5 real task ToastBread | success count (out of 20 trials): 13/20 -> 16/20 | Improved |
| fine-tune visual encoder instead of freezing it (ablation) | DP3+FVP · UR5 real task PlaceBottle | success count (out of 20 trials): 9/20 -> 20/20 | Improved |
| fine-tune visual encoder instead of freezing it (ablation) | DP3+FVP · UR5 real task ToastBread | success count (out of 20 trials): 7/20 -> 16/20 | Improved |
| use 1 history frame instead of 4 (ablation) | DP3+FVP · UR5 real task PlaceBottle | success count (out of 20 trials): 14/20 -> 20/20 | Improved |
| use 1 history frame instead of 4 (ablation) | DP3+FVP · UR5 real task PickSquare | success count (out of 20 trials): 15/20 -> 20/20 | Improved |
| pre-train DP3 encoder with FVP instead of 2D R3M encoder | DP3 · Real tasks avg (UR5, 6 tasks, Diffusion Policy) | success count (out of 20 trials): 12.5/20 -> 16.4/20 | Improved |
| pre-train DP3 encoder with FVP instead of MVP | DP3 · UR5 real task FlipCup | success count (out of 20 trials): 17/20 -> 16/20 | Hurt |
| add 3D point cloud input to RDT-1B | RDT-1B · RDT-1B generalization tasks avg | success count (out of 20 trials): 6/20 -> 8.25/20 | Improved |
| add FVP pre-training of the 3D encoder to RDT-1B with point cloud | RDT-1B · RDT-1B generalization tasks avg | success count (out of 20 trials): 8.25/20 -> 10/20 | Improved |
| add FVP pre-training to RDT-1B with point cloud | RDT-1B · RDT-1B real task PickSquare | success count (out of 20 trials): 14/20 -> 18/20 | Improved |
| add FVP pre-training to RDT-1B with point cloud | RDT-1B · RDT-1B real task PutBox | success count (out of 20 trials): 9/20 -> 9/20 | No clear effect |
Closest papers by meaning
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.92)
- Pre-training Auto-regressive Robotic Models with 4D Representations (ICML 2025 · similarity 0.91)
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations (ICML 2025 · similarity 0.90)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.90)
- 3D-MVP: 3D Multiview Pretraining for Manipulation (CVPR 2025 · similarity 0.89)
- DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
