Research map / Detection and segmentation / Tracking
DeTrack: In-model Latent Denoising Learning for Visual Object Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeTrack, a visual object tracker that treats bounding-box prediction as in-model latent denoising, with a denoising ViT whose stacked blocks each remove part of the box noise in a single forward pass, plus trajectory and visual memories.
Problem
Visual object tracking box prediction: image-feature regression ignores positional priors and autoregressive coordinate models train only on boxes in the training set, hurting generalization to unseen data; directly applying diffusion would be too slow for real-time tracking.
Approach
Injects Gaussian noise into ground-truth boxes during training and decomposes the iterative denoising process into the blocks of a single ViT forward pass (each denoising block predicts and subtracts noise from the box embedding, conditioned on search and template embeddings), combined with trajectory memory and collaboratively updated visual memory.
Key results
- DeTrack_384 reaches 77.9 AO on GOT-10k (GOT-10k-only training), above ARTrack_384 at 75.5 and SeqTrack_384 at 74.8.
- DeTrack_384 reaches 72.9 AUC on LaSOT, versus 72.6 for ARTrack_384 and 71.5 for SeqTrack_384.
- DeTrack_256 runs at 42 FPS on an RTX 3090 with 53.0 GFLOPs using a single forward pass, versus 8 FPS for 96-step multi-pass denoising.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DeTrack_384 instead of ARTrack_384 | visual object tracking · GOT-10k test | AO: 75.5 -> 77.9 | Helped |
| use DeTrack_384 instead of ARTrack_384 | visual object tracking · LaSOT_ext | AUC: 51.9 -> 53.6 | Helped |
| use a single forward pass instead of multiple forward passes (ablation) | DeTrack · GOT-10k test | AO: 75.9 -> 77.1 | Helped |
| use a single forward pass instead of multiple forward passes (ablation) | DeTrack · GOT-10k test | speed: 29FPS -> 42FPS | Helped |
| add NoisePred to the denoising block (ablation) | DeTrack · GOT-10k test | AO: 75.1 -> 77.1 | Helped |
| add denoising attention to the denoising block (ablation) | DeTrack · GOT-10k test | AO: 74.0 -> 75.1 | Helped |
| predict noise layer by layer instead of predicting total noise (ablation) | DeTrack · GOT-10k test | AO: 75.2 -> 77.1 | Helped |
| add compound visual and trajectory memories (ablation) | DeTrack · LaSOT | AUC: 70.2 -> 70.2 | Helped |
| increase visual memory length from 1 to 3 frames (ablation) | DeTrack · LaSOT | AUC | Mixed |
| increase trajectory memory length from 1 to 7 boxes (ablation) | DeTrack · LaSOT | AUC | Helped |
| set the visual-memory IoU threshold to 0.85 (ablation) | DeTrack · LaSOT | AUC | Hurt |
| use multiple forward passes instead of a single forward pass (ablation) | DeTrack with ViT-Small · GOT-10k test | AO: 69.1 -> 69.4 | Mixed |
Surprising findings
- Multiple forward passes of the denoising process did not outperform the single-forward-pass version on GOT-10k with ViT-B, and the paper reports that with ViT-Small the 48-step variant is best.
- Predicting total noise at once instead of layer-by-layer lowered AO by 1.9 points on GOT-10k.
Limitations
- Struggles to recover targets under object occlusion and out-of-view conditions, as the paper admits.
- Ablation on the Softmax and IoU thresholds for visual memory updating uses hand-tuned values selected on LaSOT.
Experimental setup
Datasets: COCO GOT-10k TrackingNet LaSOT
Benchmarks: AVisT GOT-10k LaSOT LaSOT_ext
Metrics: AUC OP50 OP75 AO SR0.5 SR0.75 P_Norm Precision FPS FLOPs
Baselines: SeqTrack ARTrack OSTrack ROMTrack STARK Mixformer TransT DiffusionTrack
Builds on: DDPM DiffusionDet ARTrack OSTrack MAE
Closest papers by meaning
- DiffusionTrack: Diffusion Model for Multi-Object Tracking (AAAI 2024 · similarity 0.91)
- DiffusionTrack: Point Set Diffusion Model for Visual Object Tracking (CVPR 2024 · similarity 0.90)
- DINTR: Tracking via Diffusion-based Interpolation (NeurIPS 2024 · similarity 0.88)
- Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision (NeurIPS 2025 · similarity 0.88)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.86)
- Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.