Research map / 3D vision / Human pose estimation
HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HOT3D, a large public egocentric multi-view dataset of hand and object 3D poses recorded with Aria and Quest 3 headsets, and benchmarks multi-view vs single-view methods for hand tracking, object pose estimation, and in-hand object lifting.
Problem
Existing hand-object datasets lack multi-view, hardware-synchronized egocentric video captured with real AR/VR headsets and accurate 3D hand and object ground truth, limiting research on multi-view egocentric 3D perception.
Approach
A large, mocap-annotated multi-view egocentric dataset recorded with real headsets (Aria, Quest 3), plus simple multi-view extensions (FoundPose, stereo DINOv2 lifting) shown to outperform single-view variants.
Key results
- Two-view UmeTrack trained on UmeTrack+HOT3D reaches 10.9 MKPE on HOT3D vs 15.4 single-view, a 41% improvement in two-view mode.
- Multi-view FoundPose improves recall at 5cm/5° from 25.2% to 33.8% on HOT3D-Aria (3 views vs 1).
- StereoMatch reaches 76.4% recall at 5cm on HOT3D-Quest3 (2 views) vs 14.3% for MonoDepth on Aria single-view at 5cm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train UmeTrack hand tracker on combined UmeTrack + HOT3D-Quest3 data instead of one dataset | UmeTrack hand tracker · HOT3D and UmeTrack hand tracking | MKPE (mm), single view: HOT3D 24.2 -> 15.4 (UmeTrack-only vs combined); UmeTrack 13.6 -> 13.4 | Helped |
| Use two-view input instead of single view for hand tracking (ablation) | UmeTrack hand tracker trained on combined data · HOT3D and UmeTrack hand tracking | MKPE (mm): HOT3D 15.4 -> 10.9; UmeTrack 13.4 -> 9.5 (about 41% improvement in two-view mode) | Helped |
| Two-view input for hand tracker trained only on HOT3D-Quest3 (cross-headset) (ablation) | UmeTrack hand tracker trained on HOT3D-Quest3 · UmeTrack test set | MKPE (mm): 23.7 -> 30.3 (single to two view) | Hurt |
| Multi-view extension of FoundPose (two or three views instead of one) (ablation) | FoundPose 6DoF object pose estimation · HOT3D-Quest3 and HOT3D-Aria | Recall at 5cm, 5 deg: Quest3 28.9 -> 36.9; Aria 25.2 -> 33.8 (8-12 points higher recall across thresholds) | Helped |
| Add predicted depth (Depth Anything V2) channel to Mask R-CNN for in-hand segmentation (ablation) | Mask R-CNN in-hand object segmentation · HOT3D-Aria and HOT3D-Quest3 | mIoU (Either hand): Aria 47.1 (RGB MRCNN) -> 55.2; Quest3 37.8 (RGB MRCNN) -> 54.7 | Helped |
| Multi-view stereo matching of DINOv2 features instead of monocular depth for 3D lifting of in-hand objects (ablation) | 3D lifting of in-hand objects · HOT3D-Aria with predicted masks | Recall at 5cm: MonoDepth 11.1 -> StereoMatch 42.6 | Helped |
| Replace hand-proxy baseline with dedicated stereo lifting for in-hand objects | 3D lifting of in-hand objects · HOT3D-Aria, GT masks | Recall at 10cm: HandProxy 13.5 -> StereoMatch 86.2 (at 10cm) | Helped |
| Off-the-shelf EgoHOS segmentation on HOT3D instead of its own dataset | EgoHOS in-hand segmentation · EgoHOS dataset vs HOT3D-Quest3 monochrome | mIoU (Either hand): Quest3 frames show ~50% lower accuracy; mIoU 52.8 on EgoHOS dataset (Both) vs 24.8 on HOT3D-Quest3 (Both) | Hurt |
Surprising findings
- Hand-tracker models trained on one dataset degrade sharply on the other, and two-view mode can be worse than single-view when trained only on the other headset's data, due to camera-configuration overfitting.
Limitations
- Ground-truth test annotations are only accessible via evaluation servers, and the dataset is restricted to rigid objects.
- Multi-view MonoDepth comparison is unavailable for Quest 3 because it lacks SLAM observations.
- Baselines are simple extensions rather than state-of-the-art multi-view methods.
Experimental setup
Datasets: HOT3D UmeTrack EgoHOS proprietary Aria in-hand object dataset
Benchmarks: HOT3D UmeTrack test set EgoHOS
Metrics: MKPE recall rate mIoU
Baselines: UmeTrack single-view FoundPose single-view EgoHOS MonoDepth HandProxy
Builds on: UmeTrack FoundPose DINOv2 Mask R-CNN OSNOM
Closest papers by meaning
- MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generation (NeurIPS 2025 · similarity 0.89)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.88)
- Instance Tracking in 3D Scenes from Egocentric Videos (CVPR 2024 · similarity 0.88)
- HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction (NeurIPS 2025 · similarity 0.88)
- EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision (CVPR 2025 · similarity 0.87)
- ComPose: When to Trust Hands for Object Pose Tracking (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.