Research map / Video / Video segmentation and tracking
Dense Optical Tracking: Connecting the Dots
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DOT, which unifies point tracking and optical flow by tracking a sparse set of points, interpolating nearest-neighbor flow and visibility, and refining them with a RAFT-like learned flow module for fast dense tracking.
Problem
Dense tracking of every pixel across video frames is too slow with existing point trackers, and optical flow methods struggle with long-range motion and occlusions.
Approach
Use a small set of off-the-shelf point tracks, sampled preferentially at motion boundaries, to initialize dense flow and visibility by nearest-neighbor interpolation, then refine them with a RAFT-style optical flow network that handles occlusions and is trained on synthetic Kubric data.
Key results
- On the CVO Extended set, DOT reduces EPE by about 4x versus optical flow baselines and is at least two orders of magnitude faster than dense point tracking (e.g., 5.2 s vs 1645 s for CoTracker per video).
- On the TAP RGB-Stacking benchmark, DOT achieves over 13% relative increase in AJ compared to single-point methods (77.1 vs 65.4 AJ on RGB-S. First).
- On CVO Final with 1024 CoTracker tracks, DOT refinement gives 10.6% relative EPE improvement over the input tracks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add track visibility information (ablation) | DOT · CVO Final | EPE (all): 1.46 -> 1.43 | Helped |
| add motion-based track sampling (ablation) | DOT · CVO Final | EPE (all): 1.50 -> 1.43 | Helped |
| use patch size 4 instead of 8 (ablation) | DOT · CVO Final | EPE (all): 1.60 -> 1.43 | Helped |
| add in-domain training (ablation) | DOT · CVO Final | EPE (all): 1.90 -> 1.43 | Helped |
| add track-based estimates (ablation) | DOT · CVO Final | EPE (all): 2.88 -> 1.43 | Helped |
| add optical-flow refinement (ablation) | DOT · CVO Final | EPE (all): 3.19 -> 1.43 | Helped |
| use DOT instead of CoTracker single-point tracking | dense tracking on DAVIS (Strided) · DAVIS (Strided) | AJ: 64.8 -> 65.9 | Helped |
| use DOT instead of CoTracker single-point tracking | dense tracking on RGB-Stacking (First) · RGB-Stacking (First) | AJ: 65.4 -> 77.1 | Helped |
| use DOT instead of TAPIR | dense tracking on Kinetics (First) · Kinetics (First) | AJ: 49.6 -> 48.4 | Hurt |
Surprising findings
- Feature-matching correspondences (SIFT, DISK, ALIKED, SuperPoint) perform much worse than point-tracking-derived tracks as DOT input, because matches are concentrated on the background.
Limitations
- Fails under extreme occlusions, fast motions, or rapid appearance changes.
- Performance depends on the quality of the input point tracker; errors in tracks propagate to the dense output.
Experimental setup
Datasets: MOVi-F Train (Kubric) CVO Train CVO Test (Clean, Final) TAP benchmark (DAVIS, RGB-Stacking, Kinetics)
Benchmarks: CVO TAP
Metrics: end point error (EPE) IoU of occluded regions inference time occlusion accuracy average jaccard position accuracy
Baselines: RAFT GMA AccFlow MFT OmniMotion PIPs++ TAPIR CoTracker TAP-Net PIPs
Builds on: RAFT CoTracker TAP-Net CVO benchmark Kubric
Closest papers by meaning
- FlowTrack: Revisiting Optical Flow for Long-Range Dense Tracking (CVPR 2024 · similarity 0.90)
- TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation (NeurIPS 2024 · similarity 0.88)
- SpatialTracker: Tracking Any 2D Pixels in 3D Space (CVPR 2024 · similarity 0.87)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.87)
- Tracking Everything Everywhere across Multiple Cameras (AAAI 2025 · similarity 0.85)
- TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.