Research map / 3D vision / Human pose estimation
RGBD Objects in the Wild: Scaling Real-World 3D Object Learning from RGB-D Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces WildRGB-D, a large-scale category-level RGB-D object video dataset captured with iPhones around objects in 360 degrees, and benchmarks four 3D tasks on it.
Problem
Existing real-world object datasets are mostly RGB-only or lack full 360-degree coverage and real-scale depth, limiting 3D object learning and RGB-D downstream tasks.
Approach
Collect ~8,500 objects and ~20K RGB-D videos across 46 categories using an iPhone depth sensor in cluttered real scenes (single-object, multi-object, and hand-object setups), with real-scale SLAM camera poses, object masks, and aggregated point clouds.
Key results
- Mip-NeRF 360 with masks reaches 35.60 PSNR and Instant-NGP 35.65 PSNR in single-scene NVS on WildRGB-D.
- Depth supervision improves Instant-NGP from 23.67 to 24.60 PSNR and cuts depth MAE from 0.366 to 0.108.
- Neusfacto with RGBD reaches 25.83 chamfer distance versus 88.92 with RGB-only on surface reconstruction (Instant-NGP RGBD 28.46 vs RGB 45.91).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use object masks during NVS training and evaluation (ablation) | NeRF-based single-scene NVS methods · WildRGB-D single-scene NVS | PSNR: 23.03/1.50 -> 34.65/4.44; 23.84/1.60 -> 35.60/4.51; 23.67/2.07 -> 35.65/5.20 | Helped |
| Add depth supervision (ablation) | Instant-NGP · WildRGB-D single-scene NVS | PSNR: 23.67/2.07 -> 24.60/2.13 | Helped |
| Add depth supervision (ablation) | Pixel-NeRF · WildRGB-D cross-scene NVS | PSNR: 18.53/1.21 -> 19.10/1.21 | Helped |
| Add extra depth-loss supervision (ablation) | MVSNeRF · WildRGB-D cross-scene NVS | PSNR: 18.43/1.30 -> 18.44/1.29 | No clear effect |
| Use depth priors (ablation) | Instant-NGP and Neusfacto · WildRGB-D object surface reconstruction | Chamfer distance: 45.91/64.01 -> 28.46/29.28; 88.92/89.94 -> 25.83/34.07 | Helped |
| Train self-supervised on WildRGB-D instead of Wild6D training data (ablation) | Category-level 6D pose estimation · Wild6D test set, bottle | IOU0.5 5 deg. 2cm: 71.3 -> 34.1 | Hurt |
| Train self-supervised on WildRGB-D instead of Wild6D training data (ablation) | Category-level 6D pose estimation · Wild6D test set, bowl | IOU0.5 5 deg. 2cm: 66.1 -> 33.8 | Hurt |
| Train on Wild6D plus WildRGB-D rather than Wild6D alone (ablation) | Category-level 6D pose estimation · Wild6D test set, bottle | IOU0.5 5 deg. 2cm: 71.3 -> 71.9 | Helped |
| Train on Wild6D plus WildRGB-D rather than Wild6D alone (ablation) | Category-level 6D pose estimation · Wild6D test set, mug | IOU0.5 5 deg. 2cm: 0.0 -> 0.0 | No clear effect |
Surprising findings
- Self-supervised 6D pose training on WildRGB-D alone is worse on some Wild6D metrics than training on Wild6D, attributed to differing camera rotation distributions.
- Geometry quality (depth MAE) is not highly correlated with rendering visual quality in cross-scene NVS.
Limitations
- No object 6D pose annotations, requiring further crowdsourcing.
- Self-supervised 6D pose transfer to Wild6D is mixed, with some metrics decreasing out-of-distribution.
- SLAM fails on some videos, so about 9% of videos were excluded.
Experimental setup
Datasets: WildRGB-D Wild6D
Benchmarks: WildRGB-D Wild6D
Metrics: PSNR SSIM LPIPS MAE rotation error chamfer distance IOU 5 deg 2cm pose accuracy
Baselines: NeRF Mip-NeRF 360 Instant-NGP Pixel-NeRF MVSNeRF IBRNet RelPose RelPose++ Neusfacto
Builds on: NeRF Instant-NGP RelPose++ Neusfacto
Closest papers by meaning
- RGB-Only Supervised Camera Parameter Optimization in Dynamic Scenes (NeurIPS 2025 · similarity 0.87)
- From an Image to a Scene: Learning to Imagine the World from a Million 360° Videos (NeurIPS 2024 · similarity 0.86)
- GIVEPose: Gradual Intra-class Variation Elimination for RGB-based Category-Level Object Pose Estimation (CVPR 2025 · similarity 0.86)
- Hand-held Object Reconstruction from RGB Video with Dynamic Interaction (CVPR 2025 · similarity 0.85)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.85)
- Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.