Research map / 3D vision / Depth and stereo
TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TrackingWorld, a pipeline that densifies sparse 2D tracks with a DELTA-style upsampler applied to every frame, then uses an optimization-based bundle adjustment to lift them into world-centric dense 3D tracks with camera poses.
Problem
Monocular 3D tracking methods fail to separate camera motion from foreground dynamic motion and cannot densely track newly emerging objects in intermediate frames.
Approach
Applies a sparse-to-dense 2D track upsampler to every frame (discarding redundant tracks in already-covered regions), then estimates world-centric camera poses and 3D tracks via a staged optimization that treats all initially static regions as potentially dynamic, with an as-static-as-possible offset term to absorb dynamic background motion.
Key results
- On Sintel camera pose, Ours (DELTA) reaches ATE 0.088 vs 0.116 for Uni4D* (ATE lower is better).
- Dense tracking depth on Sintel improves Abs Rel to 0.218 with delta<1.25 of 73.3 using UniDepth, versus 0.636 / 63.1 for raw DELTA with UniDepth.
- Upsampler on CoTrackerV3 cuts average time from 3.00 to 0.25 min on CVO-Clean while lowering EPE from 1.45 to 1.24.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TrackingWorld optimization with DELTA tracks instead of DELTA depth tracking alone | dense 3D tracking · Sintel | Abs Rel: 0.636 -> 0.218 | Helped |
| use TrackingWorld optimization with DELTA tracks instead of DELTA depth tracking alone | dense 3D tracking · Bonn | Abs Rel: 0.153 -> 0.058 | Helped |
| use TrackingWorld optimization with DELTA tracks instead of DELTA depth tracking alone | dense 3D tracking · TUM-D | Abs Rel: 0.178 -> 0.084 | Helped |
| replace UniDepth with ZoeDepth (ablation) | TrackingWorld · Sintel | ATE: 0.088 -> 0.093 | Hurt |
| replace UniDepth with Depth Pro (ablation) | TrackingWorld · Sintel | ATE: 0.088 -> 0.101 | Hurt |
| replace VLM + GroundingSAM masks with Segment Any Motion (ablation) | TrackingWorld · Sintel | Abs Rel: 0.218 -> 0.224 | Hurt |
| add the 2D upsampler (ablation) | CoTrackerV3 · CVO-Clean | EPE: 1.45 -> 1.24 | Helped |
| add the 2D upsampler (ablation) | CoTrackerV3 · CVO-Clean | Avg. Time (min): 3.00 -> 0.25 | Helped |
| enable static-track downsampling and interpolation (ablation) | TrackingWorld optimization · Sintel | T (min): 60 -> 8 | Helped |
| enable static-track downsampling and interpolation (ablation) | TrackingWorld optimization · Sintel | ATE: 0.089 -> 0.088 | No clear effect |
| add the filtering mechanism for redundant and small track regions (ablation) | TrackingWorld · Sintel | ATE: 0.105 -> 0.088 | Helped |
| track every frame instead of tracking only from the first frame (ablation) | TrackingWorld · Sintel | camera pose estimation | Helped |
Limitations
- Relies on several auxiliary foundation models (2D tracker, depth, dynamic masks), adding overhead and quality dependence.
- Pipeline is optimization-based and not feed-forward, limiting efficiency.
- Drift accumulation and pairwise approaches in feed-forward alternatives motivate future joint processing of all frames.
Experimental setup
Datasets: Sintel Bonn TUM-D ADT Panoptic Studio CVO DAVIS
Benchmarks: Sintel Bonn TUM-D ADT Panoptic Studio CVO DAVIS
Metrics: ATE RTE RRE Abs Rel delta<1.25 AJ APD_3D OA EPE IoU
Baselines: DROID-SLAM DPVO COLMAP Robust-CVD DUSt3R MonST3R Align3R Uni4D
Builds on: DELTA CoTrackerV3 Uni4D UniDepth
Closest papers by meaning
- DELTA: DENSE EFFICIENT LONG-RANGE 3D TRACKING FOR ANY VIDEO (ICLR 2025 · similarity 0.90)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.90)
- SpatialTracker: Tracking Any 2D Pixels in 3D Space (CVPR 2024 · similarity 0.89)
- TAPVid-3D: A Benchmark for Tracking Any Point in 3D (NeurIPS 2024 · similarity 0.89)
- TAPIP3D: Tracking Any Point in Persistent 3D Geometry (NeurIPS 2025 · similarity 0.89)
- Lifting Motion to the 3D World via 2D Diffusion (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.