Research map / Video / Video segmentation and tracking
FlowTrack: Revisiting Optical Flow for Long-Range Dense Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends dense optical flow to long-range dense point tracking by chaining confident flow, and switching to a transformer-based error compensation module (conditioned on flow-derived motion priors) when flow is unreliable.
Problem
Long-range dense point tracking trades off spatial density against temporal range: dense optical flow is spatially dense but drifts over long spans, while long-range point trackers are sparse and expensive.
Approach
Chain confident optical flow for efficient dense tracking, and use a transformer error compensation module that refines drifted or occluded points, conditioned on a motion encoder that processes flow-chained trajectories as spatio-temporal motion priors.
Key results
- On TAPVid-DAVIS at 256x256, FlowTrack reaches AJ 63.2 and average position accuracy 76.3, versus TAPIR at 61.3 and 73.6.
- FlowTrack achieves TC 0.53 on DAVIS (256x256) and 0.50 at 384x512, the best temporal coherence among compared methods.
- For dense tracking of 10^5 points, FlowTrack takes 145.37 s versus 284.69 s for TAPIR on the 50-frame DAVIS sequence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the error compensation module to RAFT flow chaining | FlowTrack · TAP-Vid DAVIS, 256×256 | AJ: 61.3 -> 63.2 | Helped |
| add the error compensation module to RAFT flow chaining | FlowTrack · TAP-Vid DAVIS, 256×256 | TC: 1.01 -> 0.53 | Helped |
| add the error compensation module to RAFT flow chaining | FlowTrack · TAP-Vid Kinetics | AJ: 57.2 -> 56.4 | Hurt |
| add the error compensation module to RAFT flow chaining | FlowTrack · TAP-Vid RGB-Stacking | AJ: +3.5 | Helped |
| add motion prior from optical-flow trajectories (ablation) | FlowTrack · TAP-Vid DAVIS | AJ: 60.5 -> 63.2 | Helped |
| add motion prior from optical-flow trajectories (ablation) | FlowTrack · TAP-Vid DAVIS | < δx_avg: 73.8 -> 76.3 | Helped |
| add motion prior from optical-flow trajectories (ablation) | FlowTrack · TAP-Vid DAVIS | OA: 88.7 -> 89.2 | Helped |
| add matching module for error compensation (ablation) | FlowTrack · TAP-Vid DAVIS | AJ: 38.2 -> 60.5 | Helped |
| add matching module for error compensation (ablation) | FlowTrack · TAP-Vid DAVIS | OA: 59.5 -> 88.7 | Helped |
| use FlowFormer instead of RAFT as the optical flow model (ablation) | FlowTrack · TAP-Vid DAVIS | AJ: 63.2 -> 62.4 | Hurt |
| use GMFlow instead of RAFT as the optical flow model (ablation) | FlowTrack · TAP-Vid DAVIS | AJ: 63.2 -> 57.7 | Hurt |
| use GMFlow at its training resolution instead of RAFT (ablation) | FlowTrack · TAP-Vid DAVIS | AJ: 63.2 -> 62.5 | Hurt |
Surprising findings
- Removing the motion prior drops DAVIS AJ from 63.2 to 60.5, and removing the matching module drops it further to 38.2, showing both components matter.
- The method underperforms TAPIR on TAP-Vid-Kinetics, which the authors attribute to scene cuts making flow-derived motion unreliable.
Limitations
- Falls short of TAPIR on TAP-Vid-Kinetics, where scene cuts make flow-derived motion unreliable.
- Single-point inference is slower than TAPIR due to motion encoder overhead for small numbers of query points.
- Trained only on synthetic Kubric MOVi-F data.
Experimental setup
Datasets: Kubric MOVi-F
Benchmarks: TAP-Vid DAVIS TAP-Vid Kinetics TAP-Vid RGB-Stacking
Metrics: AJ average position accuracy occlusion accuracy temporal coherence inference time
Baselines: PIPs CoTracker COTR RAFT TAP-Net OmniMotion TAPIR FlowFormer GMFlow
Builds on: RAFT TAPIR CoTracker PIPs OmniMotion
Closest papers by meaning
- Dense Optical Tracking: Connecting the Dots (CVPR 2024 · similarity 0.90)
- StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences (NeurIPS 2024 · similarity 0.88)
- EDCFlow: Exploring Temporally Dense Difference Maps for Event-based Optical Flow Estimation (CVPR 2025 · similarity 0.87)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.86)
- DELTA: DENSE EFFICIENT LONG-RANGE 3D TRACKING FOR ANY VIDEO (ICLR 2025 · similarity 0.86)
- MemFlow: Optical Flow Estimation and Prediction with Memory (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.