Research map / Vision-language and multimodal / Human and embodied modeling
RoboPEPP: Vision-Based Robot Pose and Joint Angle Estimation through Embedding Predictive Pre-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RoboPEPP, a joint-masking embedding-predictive pre-training of a ViT encoder-predictor, fine-tuned with joint-angle and keypoint heads plus confidence-based keypoint filtering and PnP for vision-based robot pose and joint angle estimation.
Problem
Estimating pose and joint angles of articulated robots with unknown joint angles from a single RGB image, where existing methods degrade under occlusion and truncation.
Approach
Pre-train a ViT encoder-predictor by masking regions around robot joints and predicting their embeddings from surrounding context (JEPA-style, joint-specific masking), then fine-tune with joint-angle and keypoint heads, random input masking, keypoint confidence filtering, and PnP.
Key results
- Achieves AUC ADD of 84.1 on Panda Synthetic DR-style test and 80.5 on Panda RS real sequence among unknown-joint methods, outperforming RoboPose and HPE on most sequences.
- Average joint angle MAE of 3.8 degrees across Panda and Kuka test sets vs 4.5 for HPE with known bounding boxes.
- Under 40% RoI occlusion RoboPEPP keeps AUC 35.1 vs 28.2 for HPE and 14.5 for RoboPose; inference 23 ms.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use joint-specific masking for embedding-predictive pre-training instead of no pre-training (ablation) | RoboPEPP · Occlusion dataset (0.4 occlusion ratio) | ADD AUC: 22.6 -> 35.1 | Helped |
| Use joint-specific masking rather than random masking during pre-training (ablation) | RoboPEPP · Occlusion dataset (0.4 occlusion ratio) | ADD AUC: 30 -> 35.1 | Helped |
| Add joint-specific masking-based pre-training (ablation) | RoboPEPP · Real-world Panda test datasets | ADD AUC | Helped |
| Add confidence-based keypoint filtering (ablation) | RoboPEPP · Panda test datasets | ADD AUC | Helped |
| Add sim-to-real self-supervised fine-tuning (ablation) | RoboPEPP · Panda real-world test datasets | ADD AUC: ∼6 points | Helped |
| Use RoboPEPP rather than HPE | robot pose estimation with unknown joint angles and detected bounding boxes · Panda AK | ADD AUC: 82.2 -> 75.3 | Hurt |
| Use RoboPEPP rather than RoboPose | robot pose estimation with unknown joint angles · Kuka DR | ADD AUC: 80.2 -> 76.2 | Hurt |
| Use RoboPEPP rather than HPE | robot pose estimation with unknown joint angles and detected bounding boxes · Baxter DR | ADD AUC: 58.8 -> 34.4 | Hurt |
| Use RoboPEPP rather than DREAM | keypoint detection · Panda test datasets (Avg.) | PCK @ 2.5 pixels: 0.43 -> 0.43 | No clear effect |
| Use RoboPEPP rather than DREAM | keypoint detection · Panda test datasets (Avg.) | PCK @ 10 pixels: 0.85 -> 0.95 | Helped |
Surprising findings
- Without ground-truth bounding boxes, HPE's joint error rises to 7.2 degrees and its AUC drops sharply (e.g., 9.8 on Baxter), showing strong sensitivity to bounding box choice.
Limitations
- HPE and RoboPose comparisons rely on re-evaluation; RoboKeyGen not directly compared because its code is unavailable.
- Panda XK HPE* evaluation was not possible due to missing weights; PCK@2.5 lower than DREAM on AK and XK.
Experimental setup
Datasets: DREAM dataset (Panda, Kuka, Baxter)
Benchmarks: DREAM dataset Panda DR Panda Photo Panda AK/XK/RS/ORB real sequences Kuka DR Kuka Photo Baxter DR custom occlusion dataset
Metrics: AUC of ADD mean absolute joint angle error (degrees) percentage of correct keypoints execution time (ms)
Baselines: HPE RoboPose DREAM-F DREAM-Q DREAM-H RoboKeyGen
Builds on: JEPA HPE DREAM EPnP
Closest papers by meaning
- Pre-training Auto-regressive Robotic Models with 4D Representations (ICML 2025 · similarity 0.85)
- DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation (NeurIPS 2026 · similarity 0.84)
- EfficientCAPER: An End-to-End Framework for Fast and Robust Category-Level Articulated Object Pose Estimation (NeurIPS 2024 · similarity 0.84)
- CAP-Net: A Unified Network for 6D Pose and Size Estimation of Categorical Articulated Parts from a Single RGB-D Image (CVPR 2025 · similarity 0.84)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.84)
- GMAP: Generalized Manipulation of Articulated Objects in Robotic Using Pre-trained Model (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.