Research map / 3D vision / 3D generation
Instance Tracking in 3D Scenes from Egocentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the IT3DEgo benchmark (HoloLens2 RGB-D egocentric videos with 3D instance annotations) and evaluates re-purposed single object trackers against a training-free SAM+DINOv2 tracking-by-detection baseline with Kalman filtering for 3D instance tracking.
Problem
Tracking object instances in 3D world coordinates from egocentric RGB-D video (for AR/VR task assistance), where objects must be recalled after leaving view, with online or pre-enrolled instance specification.
Approach
Evaluate instance tracking directly in 3D world coordinates using camera pose and depth, and propose a training-free pipeline that matches SAM proposals encoded by DINOv2 to enrolled instance images, with a Kalman filter modeling stationary object positions.
Key results
- SAM+DINOv2 achieves 1.35 m L2 error and 23.3% precision at 0.25 m under SVOE, versus 1.55 m and 21.5% for the best SOT tracker VITKT_M.
- Under MVPE, SAM+DINOv2 reaches 56.0% precision at 0.25 m and 0.67 m L2 error, while the best SOT variant (EgoSTARK template-initialized) reaches 18.9% and 1.67 m.
- Using 3D guidance raises 2D tracking AUC from 14.1 to 39.1 in MVPE and from 20.7 to 27.6 in SVOE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SAM proposals matched with DINOv2 features instead of the strongest SOT baseline | SVOE instance tracking · IT3DEgo validation | precision@0.25m: 21.5 -> 23.3 | Helped |
| Add a Kalman filter motion update (ablation) | SAM + DINOv2 · IT3DEgo SVOE validation | precision@0.25m: 23.3 -> 23.7 | Helped |
| Add a Kalman filter motion update (ablation) | SAM + DINOv2 · IT3DEgo MVPE validation | L2 error (m): 0.67 -> 0.65 | Helped |
| Use projected 3D trajectories to guide 2D proposal selection (ablation) | SAM + DINOv2 · IT3DEgo SVOE 2D tracking | AUC: 20.7 -> 27.6 | Helped |
| Use projected 3D trajectories to guide 2D proposal selection (ablation) | SAM + DINOv2 · IT3DEgo MVPE 2D tracking | AUC: 14.1 -> 39.1 | Helped |
| Increase the cosine similarity threshold (ablation) | SAM + DINOv2 · IT3DEgo validation | precision and recall | Mixed |
| Use multi-view pre-enrollment rather than single-view online enrollment | SAM + DINOv2 · IT3DEgo instance tracking | L2 error (m): 1.35 -> 0.67 | Helped |
Surprising findings
- Depth-aware RGB-D trackers perform similar to or worse than RGB-only short-term trackers, since sparse depth maps are only used as auxiliary features.
- A simple Kalman filter gives only marginal gains over the most-recent-location memory heuristic.
Limitations
- Benchmark has limited geographic and demographic diversity and few object categories, so it is only a diagnostic test rather than training data.
- Baselines all lift 2D trajectories to 3D and rely on a simple memory mechanism; more nuanced state-update models are not explored.
- Experiments are on the validation split only; test set reserved for future work.
Experimental setup
Datasets: IT3DEgo benchmark (HoloLens2 RGB-D videos, 50 recordings, 10 indoor scenes)
Benchmarks: IT3DEgo
Metrics: precision at L2 thresholds recall at L2 thresholds L2 error angular error AUC normalized precision
Baselines: ToMP MixFormer ARTrack SAMF MixForRGBD ViPT mixLT mlpLT VITKT_M EgoSTARK
Builds on: SAM DINOv2 Kalman filter COLMAP
Closest papers by meaning
- FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video (CVPR 2025 · similarity 0.89)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.88)
- HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos (CVPR 2025 · similarity 0.88)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.87)
- 3D Human Pose Perception from Egocentric Stereo Videos (CVPR 2024 · similarity 0.86)
- TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.