Research map / 3D vision / 3D generation
3D-MVP: 3D Multiview Pretraining for Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pretrains the visual encoder of the Robotic View Transformer (RVT) with multi-view masked autoencoding on rendered Objaverse point clouds, then fine-tunes it for 3D robot manipulation on RLBench and COLOSSEUM.
Problem
Existing MAE-based robotics pretraining learns only 2D image encoders, which do not suit manipulation policies that reason over explicit 3D scene representations, and such policies overfit when trained from scratch on limited demonstrations.
Approach
Split RVT's multi-view transformer into a visual encoder and action decoder, and pretrain the encoder by masking tokens across five orthogonal RGB-D virtual views rendered from Objaverse point clouds and reconstructing all views, so the encoder learns cross-view 3D-aware features before fine-tuning on manipulation.
Key results
- On RLBench 18 tasks, 3D-MVP reaches 67.5 average success rate versus 62.9 for RVT trained from scratch and 49.4 for PerAct.
- Pretraining on the full Objaverse (200K objects) gives 67.6 average success, while 3D-FRONT pretraining gives only 63.6 and a 18K-object Objaverse subset gives 65.3.
- On COLOSSEUM, 3D-MVP is more robust than RVT on most of 12 perturbations and clearly outperforms 2D pretraining methods MVP and R3M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pretrain the RVT encoder with 3D-MVP on Objaverse | RVT · RLBench | average success rate: 62.9 -> 67.5 | Helped |
| add Objaverse pretraining (ablation) | 3D-MVP on RLBench · RLBench | average success rate: 62.9 -> 67.6 | Helped |
| use Objaverse full rather than Objaverse small for pretraining (ablation) | 3D-MVP · RLBench | average success rate: 65.3 -> 67.6 | Helped |
| pretrain on Objaverse rather than 3D-FRONT (ablation) | 3D-MVP · RLBench | average success rate: 63.6 -> 67.6 | Helped |
| pretrain on RLBench point clouds rather than train from scratch (ablation) | 3D-MVP · RLBench | average success rate: 62.9 -> 67.5 | Helped |
| add pretraining on RLBench point clouds (ablation) | 3D-MVP · COLOSSEUM | generalization | Hurt |
| mask RGB channels instead of all channels (ablation) | 3D-MVP · RLBench | average success rate: 64.4 -> 67.6 | Helped |
Surprising findings
- Pretraining MAE on RLBench scenes alone reaches 67.5 success but overfits and fails to generalize to COLOSSEUM renderings.
- Masking only RGB channels performed better than masking all channels (67.5 vs 64.7 on RLBench).
Limitations
- Uses a fixed set of camera viewpoints and does not explicitly reason about occlusions or spatial relations between objects.
- Assumes quasi-static dynamics and does not handle dynamic robot-environment interactions.
- Still requires a small amount of labeled demonstration data per downstream task and cannot generalize to unseen tasks.
Experimental setup
Datasets: Objaverse 3D-FRONT RLBench COLOSSEUM Ego4D EpicKitchen 100DOH
Benchmarks: RLBench COLOSSEUM
Metrics: task success rate average success rate
Baselines: Image-BC (CNN) Image-BC (ViT) C2F-ARM-BC PerAct RVT MVP R3M
Builds on: Robotic View Transformer (RVT) masked autoencoder (MAE) MVP PerAct
Closest papers by meaning
- SUGAR: Pre-training 3D Visual Representations for Robotics (CVPR 2024 · similarity 0.87)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.87)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.87)
- DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation (NeurIPS 2025 · similarity 0.87)
- PDFactor: Learning Tri-Perspective View Policy Diffusion Field for Multi-Task Robotic Manipulation (CVPR 2025 · similarity 0.86)
- BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.