Research map / Detection and segmentation / Tracking
Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Tracktention Layer, a plug-and-play module that injects off-the-shelf point-track information via track cross-attention so image networks (Depth Anything, DDColor etc.) become temporally consistent video models.
Problem
Temporal inconsistency (flicker, artifacts) in per-frame video prediction tasks such as video depth estimation and colorization, where temporal attention and 3D convolutions struggle with large object motion and long-range dependencies.
Approach
Sample image tokens at point-track locations via attentional sampling, propagate them along each track with a temporal transformer, and splat them back to image tokens via attentional splatting; inserted as residual layers with zero-initialized output projections into existing ViT/ConvNet models.
Key results
- On the DepthCrafter benchmark, Tracktention on Depth Anything (140M params) reaches average AbsRel 0.138 and delta1.25 0.862, beating DepthCrafter (0.163 / 0.843) with far fewer parameters.
- On video colorization, Tracktention improves CDC by 22.8% and 28.3% for CIC on DAVIS and Videvo, 41.3% and 28.8% for Colorformer, and 46.5% and 45.7% for DDColor.
- On RobustMVD, the model achieves the lowest average relative error (6.2) and highest inlier ratio (41.1%), with runtime 0.9 s per sequence including about 0.4 s for tracking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Tracktention to Depth Anything | Depth Anything-Base · DepthCrafter benchmark average | AbsRel: 0.169 -> 0.138 | Helped |
| add Tracktention to Depth Anything | Depth Anything-Base · RobustMVD average | rel: 8.3 -> 6.2 | Helped |
| add Tracktention to DDColor | DDColor · DAVIS | CDC: 8.64 -> 4.62 (-46.5%) | Helped |
| add Tracktention to DDColor | DDColor · Videvo | CDC: 5.60 -> 3.04 (-45.7%) | Helped |
| add Tracktention to ColorFormer | ColorFormer · DAVIS | CDC: 8.29 -> 4.86 (-41.3%) | Helped |
| add Tracktention to ColorFormer | ColorFormer · Videvo | CDC: 4.96 -> 3.53 (-28.8%) | Helped |
| add Tracktention to CIC | CIC · DAVIS | CDC: 6.18 -> 4.77 (-22.8%) | Helped |
| add Tracktention to CIC | CIC · Videvo | CDC: 3.59 -> 2.57 (-28.3%) | Helped |
| increase training tracks from 0 to 242 (ablation) | Tracktention for depth prediction · Sintel | AbsRel: 0.325 -> 0.295 | Helped |
| increase training tracks from 0 to 242 (ablation) | Tracktention for depth prediction · ScanNet | AbsRel: 0.130 -> 0.087 | Helped |
| replace tracked points with constant tracks (ablation) | Tracktention for depth prediction · Sintel | AbsRel: 0.295 -> 0.352 | Hurt |
| replace Tracktention with 3D convolution (ablation) | Tracktention for depth prediction · Sintel | AbsRel: 0.295 -> 0.364 | Hurt |
Surprising findings
- Using a single track per video during training already improves depth accuracy over the base model on Sintel-like data, while replacing trackers with constant pseudo-tracks performs worse than using a real tracker.
- Attention across tracks in the Track Transformer was slightly worse and more expensive than per-track temporal attention.
Limitations
- Performance depends on the quality of extracted point tracks; trackers may fail when tracks are off-camera for long periods.
- Running an additional point tracker adds computational cost and complexity.
Experimental setup
Datasets: YouTube-VIS DAVIS Videvo Sintel ScanNet KITTI Bonn RobustMVD ETH3D DTU Tanks and Temples
Benchmarks: DepthCrafter video depth benchmark (Sintel, ScanNet, KITTI, Bonn) RobustMVD DAVIS Videvo
Metrics: AbsRel delta1.25 FID colorfulness CDC PSNR
Baselines: DUSt3R NVDS ChronoDepth DepthCrafter Marigold Depth Anything V2 UniDepth FAVC TCVC CIC IDC ColorFormer
Builds on: Depth Anything CoTracker3 DDColor RoPE QK-normalization
Closest papers by meaning
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.92)
- TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation (NeurIPS 2024 · similarity 0.91)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.89)
- Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation (CVPR 2025 · similarity 0.88)
- Exploring Temporally-Aware Features for Point Tracking (CVPR 2025 · similarity 0.88)
- Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.