Research map / Detection and segmentation / Tracking
TrackIME: Enhanced Video Point Tracking via Instance Motion Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TrackIME, a plug-in framework that estimates an object instance's trajectory via SAM-based semantic neighbor point tracking and aggregation, then prunes video frames around it so point trackers run without down-sampling losses, jointly improving tracking and segmentation.
Problem
Point tracking models must densely match correspondences across all frames, forcing down-sampled resolutions that lose fine detail and cause tracking failures on intricate object parts.
Approach
Identify the instance mask of the query point with SAM, track a group of semantic-neighbor points on it, aggregate their visibility-weighted velocities into an instance trajectory, prune frames around that trajectory at high resolution, re-sample occluded points via aggregated masks, and iterate progressively.
Key results
- On DAVIS first-query tracking, TAPIR+TrackIME reaches 65.3 AJ vs 57.5 for vanilla TAPIR (about 13.6% relative gain).
- J1 under DAVIS-F improves from 28.3 (CoTracker) to 35.4 with TrackIME, a relative gain of up to 25.1%.
- Zero-shot DAVIS-2017 val segmentation reaches 79.6 (J&F)m vs 78.8 for SAM-PT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TrackIME to CoTracker | CoTracker · DAVIS | AJ: 60.8 -> 64.5 | Helped |
| Add TrackIME to OmniMotion | OmniMotion · RGBStacking | AJ: 71.2 -> 71.9 | Helped |
| Add TrackIME to OmniMotion | OmniMotion · Kinetics | AJ: 51.0 -> 51.2 | Helped |
| Use TrackIME instead of SAM-PT | Zero-shot video segmentation · DAVIS-2017-val | (J&F)m: 78.8 -> 79.6 | Helped |
| Add search-space pruning based on the query trajectory (ablation) | TAPIR · DAVIS-F | J1: 23.0 -> 28.2 | Helped |
| Add trajectory aggregation from segmentation-mask points (ablation) | TrackIME with TAPIR · DAVIS-F | J1: 28.2 -> 34.0 | Helped |
| Add progressive inference when trajectory aggregation is enabled (ablation) | TrackIME with TAPIR · DAVIS-F | J1: 34.0 -> 35.4 | Helped |
| Add progressive inference without trajectory aggregation (ablation) | TrackIME with TAPIR · DAVIS-F | J1: 28.2 -> 28.3 | No clear effect |
| Vary pruning size from 1080 to 384 with K = 1 (ablation) | TrackIME with TAPIR · DAVIS-F | AJ: 62.5; 63.1; 64.0; 63.9; 62.3 | Mixed |
| Use progressive pruning sizes 960 → 384 with K = 2 (ablation) | TrackIME with TAPIR · DAVIS-F | AJ: 63.9 -> 65.3 | Helped |
| Vary the number of semantic neighbors from 128 to 2 (ablation) | TrackIME with TAPIR · DAVIS-F | AJ: 64.8; 64.8; 65.3; 64.8; 64.9; 64.9; 64.7 | No clear effect |
| Replace visibility-weighted aggregation with equal weights (ablation) | TrackIME with TAPIR · DAVIS-F | AJ: 65.3 -> 64.0 | Hurt |
Surprising findings
- Pruning alone, without the segmentation-based aggregation, already raises TAPIR's AJ from 57.5 to 62.5, and simply raising input resolution without fine-tuning degrades TAPIR's performance.
- Occlusion accuracy gains are only incremental, and pruning can remove information from other instances, creating a trade-off in occlusion modeling.
Limitations
- Relies on pre-trained point trackers trained on synthetic data (Kubric, PointOdyssey) while segmentation models are trained on real images.
- Pruning removes information from other instances, trading off occlusion modeling against search-space pruning.
- Higher total compute (about 2789 GFLOPs vs 612 for TAPIR) and multi-stage pipeline with SAM.
Experimental setup
Datasets: DAVIS Kinetics RGB-Stacking DAVIS-2017
Benchmarks: TAP-Vid DAVIS TAP-Vid Kinetics TAP-Vid RGB-Stacking DAVIS-2017 val DAVIS-2017 test-dev
Metrics: average Jaccard (AJ) delta-avg accuracy delta-1 accuracy J1 occlusion accuracy J&F mean Jm Fm
Baselines: TAPNet PIPS2 TAPIR CoTracker OmniMotion SAM-PT EntitySeg
Builds on: TAPIR SAM HQ-SAM SAM-PT
Closest papers by meaning
- Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better (CVPR 2025 · similarity 0.91)
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.90)
- Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision (NeurIPS 2025 · similarity 0.89)
- Dense Optical Tracking: Connecting the Dots (CVPR 2024 · similarity 0.88)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.88)
- SpatialTracker: Tracking Any 2D Pixels in 3D Space (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.