Research map / 3D vision / Human pose estimation
Free-Moving Object Reconstruction and Pose Estimation with Virtual Camera
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reconstructs a free-moving rigid object and its pose from a monocular RGB video without priors or segmenting, by optimizing a NeuS-based SDF with a 4-DoF pose MLP in a mask-guided virtual camera, then refining with the real camera.
Problem
Joint shape reconstruction and pose estimation of freely moving objects from monocular RGB video without scene, hand, or category priors, and without segment-wise optimization.
Approach
A mask-guided virtual camera that always points at the object center reduces the unknown object pose to 4 DoF (rotation plus distance), enabling globally optimized, segment-free progressive training of an implicit SDF representation with frame-wise pose MLP, match loss from LoFTR, periodic shape-network resets, and a final real-camera refinement via PnP.
Key results
- Mesh HD_RMSE average 3.14 on HO3D over 9 sequences, versus 4.64 for Hampali et al. and 13.41 for COLMAP, and on par with NeuS trained with ground-truth poses (2.35).
- Pose AUC_ATE average 5.9 versus 4.5 for Hampali et al. and 2.9 for COLMAP.
- Virtual camera improves AUC_ATE from 4.33 (real camera) to 5.42 and, with real-camera refinement, to 5.93.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the guided virtual camera instead of optimizing progressively with the real camera only (ablation) | Ours · HO3D | AUC_ATE: 4.33 -> 5.42 | Helped |
| Add global refinement with the real camera after virtual-camera optimization (ablation) | Ours with virtual camera · HO3D | AUC_ATE: 5.42 -> 5.93 | Helped |
| Use the guided virtual camera instead of optimizing progressively with the real camera only (ablation) | Ours · HO3D | HD_RMSE: 4.15 -> 3.90 | Helped |
| Add global refinement with the real camera after virtual-camera optimization (ablation) | Ours with virtual camera · HO3D | HD_RMSE: 3.90 -> 3.14 | Helped |
| Add the match loss (ablation) | Ours · HO3D | HD_RMSE: 4.10 -> 3.14 | Helped |
| Add periodic shape-network reset (ablation) | Ours · HO3D | HD_RMSE: 3.28 -> 3.14 | Helped |
| Use the 4D pose representation instead of the standard 6D pose representation (ablation) | Ours · HO3D | HD_RMSE: 3.27 -> 3.14 | Helped |
| Use the proposed method rather than Hampali et al. | HO3D reconstruction · HO3D (Average) | HD_RMSE (mm): 4.64 -> 3.14 | Helped |
| Use the proposed method rather than Hampali et al. | HO3D pose estimation · HO3D (Average) | AUC_ATE: 4.5 -> 5.9 | Helped |
Limitations
- Fails when parts of the object are occluded for a long time during capture.
- Inaccurate for tiny texture-less objects due to lack of features.
- Virtual-camera pose is not physically compliant and needs real-camera refinement.
Experimental setup
Datasets: HO3D egocentric sequences captured with Magic Leap 2
Benchmarks: HO3D
Metrics: HD_RMSE AUC_ATE RPE_t RPE_r
Baselines: COLMAP Ye et al. Hampali et al. UNISURF NeuS Patten et al. BARF
Builds on: NeuS BARF LoFTR COLMAP Hampali et al. segment-wise optimization
Closest papers by meaning
- Hand-held Object Reconstruction from RGB Video with Dynamic Interaction (CVPR 2025 · similarity 0.90)
- OnlineSplatter: Pose-Free Online 3D Reconstruction for Free-Moving Objects (NeurIPS 2025 · similarity 0.88)
- In-Hand 3D Object Reconstruction from a Monocular RGB Video (AAAI 2024 · similarity 0.87)
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos (CVPR 2025 · similarity 0.86)
- UNOPose: Unseen Object Pose Estimation with an Unposed RGB-D Reference Image (CVPR 2025 · similarity 0.86)
- MFOS: Model-Free & One-Shot Object Pose Estimation (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.