Research map / Video / Video generation
DINTR: Tracking via Diffusion-based Interpolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DINTR, a tracking-by-diffusion method that fine-tunes latent diffusion models online to interpolate between consecutive video frames in latent space and extracts target locations from attention maps, evaluated across point, pose, box, text and segment tracking.
Problem
Object tracking across five indication types (point, pose, box, segment, text) typically requires specialized architectures; existing diffusion-based trackers operate only on 2D coordinates and rely on extensive Gaussian noise mapping that is unnecessary for temporal correspondence.
Approach
Replace the step-wise Gaussian noise denoising process with a deterministic data-interpolation process in visual latent space that transitions from the current frame latent to the next frame latent, and extract object locations from averaged self- and cross-attention maps conditioned on the indicator.
Key results
- On TAP-Vid point tracking, DINTR reaches AJ 57.8 on Kinetics and 85.5 on Kubric, outperforming TAPIR on Kinetics and DAVIS (62.3 vs 59.8 AJ on DAVIS).
- On PoseTrack21, DINTR scores 82.5 mAP, below DiffPose (83.0) but with higher MOTA (64.9) and HOTA (55.5) than listed baselines.
- On MOT17, DINTR gets 78.0 MOTA and 63.5 HOTA, trailing DiffMOT (79.8 MOTA, 64.5 HOTA); on DAVIS segmentation it gets J&F 75.4 vs UNICORN 69.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use visual interpolation instead of reconstruction (ablation) | DINTR · TAP-Vid | AJ: 53.6 -> 57.8 (Kinetics); 80.5 -> 85.5 (Kubric); 62.0 -> 62.3 (DAVIS); 62.3 -> 65.2 (RGB-Stacking) | Helped |
| Use visual interpolation instead of reconstruction (ablation) | DINTR · PoseTrack | mAP: 77.8 -> 82.5 | Helped |
| Use visual interpolation instead of reconstruction (ablation) | DINTR · LaSOT, with text | Precision: 0.52 -> 0.60 | Helped |
| Use visual interpolation instead of reconstruction (ablation) | DINTR · VOS | J&F: 73.9 -> 75.7 | Helped |
| Use offset learning instead of linear blending (ablation) | DINTR · TAP-Vid | AJ: 27.6 -> 57.8 (Kinetics); 54.6 -> 85.5 (Kubric); 48.1 -> 62.3 (DAVIS); 55.6 -> 65.2 (RGB-Stacking) | Helped |
| Use offset learning instead of linear blending (ablation) | DINTR · PoseTrack | mAP: 59.7 -> 82.5 | Helped |
| Learn from destination latent instead of offset learning (ablation) | DINTR · TAP-Vid | AJ: 64.9 -> 85.5 (Kubric) | Hurt |
| Learn from source latent instead of offset learning (ablation) | DINTR · TAP-Vid | AJ: 63.3 -> 85.5 (Kubric) | Hurt |
| Use destination-latent guidance rather than source-latent guidance (ablation) | DINTR interpolation · TAP-Vid Kubric | AJ: 63.3; 64.9 | No clear effect |
| Increase diffusion steps from 50 to 250 (ablation) | DINTR reconstruction · Reconstruction quality | MSE: 20.5 -> 0.04 | Helped |
| Increase diffusion steps from 50 to 250 (ablation) | DINTR · VOS | J&F: 75.4 -> 76.5 | Helped |
| Increase diffusion steps from 50 to 250 (ablation) | DINTR interpolation · Inference time | time (s): 3.2 -> 14.7 | Hurt |
Surprising findings
- Interpolation-based temporal modeling outperforms reconstruction-from-noise by a clear margin, and linear blending is the weakest variant.
- The interpolation process is reported to be about 2x faster than reconstruction (3.2 s vs 6.2 s at T=50).
Limitations
- Small performance gap to motion-model methods such as DiffMOT that use 2D coordinate diffusion.
- Reconstruction variant is time-consuming, limiting practicality; the method is not real-time.
- Per-frame online fine-tuning and dependence on external detectors for first-frame initialization.
Experimental setup
Datasets: TAP-Vid PoseTrack21 LaSOT GOT DAVIS MOTS MOT17
Benchmarks: TAP-Vid PoseTrack21 LaSOT DAVIS MOTS MOT17 GOT
Metrics: AJ <delta_avg OA mAP MOTA IDF1 HOTA Precision Success J&F
Baselines: TAPIR PIPs TAP-Net RAFT COTR DiffPose CorrTrack Tracktor++ UNICORN UniTrack DiffusionTrack DiffMOT
Builds on: latent diffusion models Prompt-to-Prompt DDIM ADM
Closest papers by meaning
- DiffusionTrack: Point Set Diffusion Model for Visual Object Tracking (CVPR 2024 · similarity 0.92)
- DiffusionTrack: Diffusion Model for Multi-Object Tracking (AAAI 2024 · similarity 0.90)
- Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision (NeurIPS 2025 · similarity 0.89)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.89)
- DeTrack: In-model Latent Denoising Learning for Visual Object Tracking (NeurIPS 2024 · similarity 0.88)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.