Research map / 3D vision / 3D generation
Unsupervised Learning of Visual 3D Keypoints for Control
Boyuan Chen, Pieter Abbeel, Deepak Pathak
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 3D keypoints instead of RAD's representation | Visual RL on Meta-World · Meta-World (8 environments) | RL performance | Improved |
| Use 3D keypoints instead of visual RL baselines | Visual RL for locomotion · PyBullet Ant and colorless Ant | RL performance | Improved |
| Use 3D keypoints instead of prior visual RL methods | Scarf manipulation · Scarf manipulation | RL performance | Improved |
| Use 3D keypoints rather than baseline representations | Transfer learning · ML45 training environments | RL performance | Improved |
| Use a learned representation instead of RAD's representation | Transfer learning · ML45 test environments | transfer performance | Improved |
| Remove random crop offset (ablation) | Keypoint3D · Meta-World open door | RL performance | Hurt |
| Remove multi-view consistency loss (ablation) | Keypoint3D · Meta-World ablation environments | RL performance | Hurt |
| Use triangulation to map 2D keypoints into 3D instead of the learned 3D keypoint method (ablation) | Keypoint3D · Meta-World ablation environments | RL performance | Hurt |
| Train keypoints on random-policy data, then freeze the encoder for policy learning (ablation) | Keypoint3D · Meta-World joint-training ablation | RL performance | Hurt |
| Train keypoints on expert-policy data, then freeze the encoder for policy learning (ablation) | Keypoint3D · Meta-World joint-training ablation | RL performance | Hurt |
| Stop policy gradients from updating the encoder (ablation) | Keypoint3D · Meta-World joint-training ablation | RL performance | Hurt |
| Remove attention (ablation) | Keypoint3D · Reinforcement learning ablation environments | RL performance | Hurt |
Closest papers by meaning
- Pre-training Auto-regressive Robotic Models with 4D Representations (ICML 2025 · similarity 0.88)
- Lift3D Policy: Lifting 2D Foundation Models for Robust 3D Robotic Manipulation (CVPR 2025 · similarity 0.87)
- DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation (NeurIPS 2025 · similarity 0.86)
- Key-Grid: Unsupervised 3D Keypoints Detection using Grid Heatmap Features (NeurIPS 2024 · similarity 0.86)
- PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics (NeurIPS 2026 · similarity 0.86)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
