Research map / 3D vision / 3D generation
DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
DynaRend pretrains a triplane scene encoder with masked reconstruction and future prediction supervised by differentiable volumetric rendering of RGB, DINO/RADIO-style semantics, and depth, then fine-tunes it for keyframe action value map prediction on RLBench, Colosseum, and real-world tasks.
Problem
Robotic manipulation policies lack generalizable visual representations because existing self-supervised methods are either static 2D or 2D video-based and do not jointly capture 3D geometry, semantics, and future dynamics.
Approach
Learn a unified triplane representation by masked reconstruction of the current scene and prediction of the future keyframe scene, both supervised through differentiable volumetric rendering of RGB, semantic features, and depth, with synthetic novel views from See3D to reduce reliance on dense cameras.
Key results
- On the 18-task RLBench setting DynaRend reaches 83.2% average success rate versus 81.4% for RVT-2 and 62.9% for RVT.
- On the 71-task RLBench setting (two groups) DynaRend averages 76.6% versus 70.8% for SPA and 61.1% for single-stage RVT.
- On real-world tasks DynaRend averages 57% success on the standard setting versus 37% for RVT-2 and 3D Diffuser Actor, and 45% versus 16% for RVT-2 with distractors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain with both reconstruction and future prediction objectives instead of either objective alone (ablation) | DynaRend · RLBench 18 tasks | average success rate | Helped |
| Use future prediction instead of reconstruction alone during pretraining (ablation) | DynaRend · RLBench 18 tasks | average success rate | Helped |
| Add a moderate triplane masking ratio (ablation) | DynaRend · Downstream policy evaluation | success rate | Helped |
| Remove triplane masking or use an excessively high mask ratio (ablation) | DynaRend · Downstream policy evaluation | success rate | Hurt |
| Add RGB reconstruction loss (ablation) | DynaRend · RLBench 18 tasks | average success rate | Helped |
| Add semantic alignment loss (ablation) | DynaRend · RLBench 18 tasks | average success rate | Helped |
| Add depth supervision (ablation) | DynaRend · RLBench 18 tasks | average success rate | Helped |
| Add synthetic target-view augmentation during pretraining (ablation) | DynaRend · RLBench 18 tasks | downstream success rate | Helped |
Surprising findings
- Future prediction alone yields larger gains than reconstruction alone during pretraining, per the ablation.
- Depth supervision contributes only moderately, which the authors attribute to the triplane already encoding explicit 3D information from depth-projected point clouds.
Limitations
- Still relies on an external low-level motion planner to execute predicted keyframe actions.
- Target view augmentation is limited to keyframe viewpoints and small angular offsets (±30 degrees) because novel-view quality degrades with larger offsets.
- Real-world evaluation uses 20 rollouts per task and only five tasks.
Experimental setup
Datasets: RLBench Colosseum real-world multi-view RGB-D demonstrations
Benchmarks: RLBench-18 RLBench-71 Colosseum real-world five-task suite
Metrics: average success rate average rank inference speed
Baselines: RVT RVT-2 3D Diffuser Actor PerAct C2F-ARM-BC 3D-MVP SPA MVP VC-1 MoCov3 MAE DINOv2
Builds on: RVT GNFactor SPA RADIOv2.5 See3D
Closest papers by meaning
- DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation (NeurIPS 2026 · similarity 0.91)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.90)
- Localizing, Structuring, and Rendering: Bridging 3D and 2D Vision-Language-Action Models for Robotic Manipulation (CVPR 2026 · similarity 0.89)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.89)
- PDFactor: Learning Tri-Perspective View Policy Diffusion Field for Multi-Task Robotic Manipulation (CVPR 2025 · similarity 0.88)
- AffordDP: Generalizable Diffusion Policy with Transferable Affordance (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.