Research map / 3D vision / Human pose estimation
CAP-Net: A Unified Network for 6D Pose and Size Estimation of Categorical Articulated Parts from a Single RGB-D Image
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CAP-Net, a single-stage RGB-D network that jointly predicts part semantic labels, centroid offsets, and NPCS maps for 6D pose and size of categorical articulated parts, and introduces the RGBD-Art photorealistic synthetic dataset.
Problem
Category-level 6D pose and size estimation of articulated parts for robotic manipulation, where existing multi-stage segment-then-pose pipelines lose global context and suffer from sim-to-real gaps and depth noise.
Approach
A unified single-shot network fuses frozen SAM2 and FeatUp RGB features point-wise with the depth point cloud and jointly predicts semantic class, instance centroid offsets, and NPCS maps; instances are obtained by DBSCAN clustering of centroids and poses by RANSAC plus Umeyama alignment.
Key results
- On RGBD-Art, CAP-Net reaches rotation error 10.39 deg and A5 33.91% versus GAPartNet's 83.3 deg and 0.71%.
- Part segmentation AP50 of 53.58 on seen categories versus 11.35 for GAPartNet, and 19.38 vs 6.59 on unseen categories.
- Real robot experiments succeed in 28/30 trials versus 5/30 for GAPartNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CAP-Net instead of GAPartNet | articulated part segmentation · RGBD-Art seen categories | AP50 (%): 11.35 -> 53.58 | Helped |
| Use CAP-Net instead of GAPartNet | articulated part segmentation · RGBD-Art unseen categories | AP50 (%): 6.59 -> 19.38 | Helped |
| Use CAP-Net instead of GAPartNet | articulated part pose estimation · RGBD-Art | rotation error Re: 83.3 -> 10.39 | Helped |
| Use CAP-Net instead of GAPartNet | robot manipulation · Real-world robot manipulation | success rate: 5/30 -> 28/30 | Helped |
| Train the semantic, instance-offset, and NPCS heads jointly rather than independently (ablation) | CAP-Net · RGBD-Art | rotation error Re: 15.30 -> 10.39 | Helped |
| Train the semantic, instance-offset, and NPCS heads jointly rather than independently (ablation) | CAP-Net · RGBD-Art | 3D mIoU: 51.75 -> 56.23 | Helped |
| Add SAM features to the PointNet++ backbone (ablation) | CAP-Net · RGBD-Art | rotation error Re: 21.30 -> 12.20 | Helped |
| Add FeatUp features to the PointNet++ backbone (ablation) | CAP-Net · RGBD-Art | rotation error Re: 21.30 -> 20.03 | Helped |
| Add FeatUp features to the PointNet++ backbone (ablation) | CAP-Net · RGBD-Art | 3D mIoU: 41.09 -> 43.52 | Helped |
| Add DINOv2 features to the PointNet++ backbone (ablation) | CAP-Net · RGBD-Art | rotation error Re: 21.30 -> 25.12 | Hurt |
| Add DINOv2 features to the PointNet++ backbone (ablation) | CAP-Net · RGBD-Art | 3D mIoU: 41.09 -> 40.10 | No clear effect |
Surprising findings
- Two-stage GAPartNet's rotation error is inflated largely because strict evaluation removes its 180-degree mirror symmetry tolerance.
Limitations
- Relies on depth for pose fitting; missing surface points degrade accuracy.
- Low absolute AP50 on unseen small parts (e.g., 0.925 for round fixed handle).
Experimental setup
Datasets: RGBD-Art GAPartNet PartNet-Mobility OPD MultiScan
Benchmarks: RGBD-Art OPD MultiScan
Metrics: AP50 AP rotation error translation error scale error 3D mIoU A5 A10 robot success rate
Baselines: PointGroup SoftGroup AutoGPart GAPartNet
Builds on: NOCS PointNet++ GAPartNet FeatUp SAM2
Closest papers by meaning
- SchemaPose: RGB-Based Category-Level Pose Estimation with Parametric Category Schema (NeurIPS 2026 · similarity 0.89)
- R^2-Art: Category-Level Articulation Pose Estimation from Single RGB Image via Cascade Render Strategy (AAAI 2025 · similarity 0.88)
- GIVEPose: Gradual Intra-class Variation Elimination for RGB-based Category-Level Object Pose Estimation (CVPR 2025 · similarity 0.88)
- HouseCat6D - A Large-Scale Multi-Modal Category Level 6D Object Perception Dataset with Household Objects in Realistic Scenarios (CVPR 2024 · similarity 0.87)
- 6D Object Pose Tracking in Internet Videos for Robotic Manipulation (ICLR 2025 · similarity 0.87)
- SingRef6D: Monocular Novel Object Pose Estimation with a Single RGB Reference (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.