Research map / Detection and segmentation / Tracking
DreamTrack: Dreaming the Future for Multimodal Visual Object Tracking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates temporal visual object tracking as a History-to-Future process: a GRU-based Future Dreaming module predicts future search-region features, and a multimodal head forecasts several possible target trajectories over the current and next two frames.
Problem
Visual object tracking with template-matching trackers generalizes poorly under severe appearance deformation and background variation in long-term tracking, and existing temporal trackers only pass historical outputs rather than learning environmental dynamics.
Approach
Borrow the world-model dreaming idea: target queries cache temporal dynamics via a GRU and imagine future observation features (self-supervised by KL and reconstruction losses on future frames), with temporal self/cross-attention refinement and multimodal (3-mode) trajectory prediction weighted by probabilities.
Key results
- DreamTrack-L384 reaches 76.6% SUC on LaSOT, surpassing ARTrackV2-L384 by 3.0% and LoRAT-L378 by 1.5%.
- DreamTrack256 runs at 139 FPS on an RTX2080Ti while reaching 73.8% SUC on LaSOT.
- Ablation: Future Dreaming module adds 2.4% SUC on LaSOT over the baseline (71.7 vs 69.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DreamTrack-L384 instead of the strongest listed baseline | visual object tracking · LaSOT test | SUC: 75.1 -> 76.6 | Helped |
| use DreamTrack-L384 instead of the strongest listed baseline | visual object tracking · LaSOText | SUC: 56.6 -> 57.1 | Helped |
| use DreamTrack-L384 instead of the strongest listed baseline | visual object tracking · GOT-10k test | AO: 79.5 -> 79.9 | Helped |
| use DreamTrack-L384 instead of the strongest listed baseline | visual object tracking · TrackingNet test | SUC: 86.1 -> 87.9 | Helped |
| use DreamTrack-L384 instead of the strongest listed baseline | visual object tracking · TNL2K test | SUC: 62.3 -> 62.9 | Helped |
| use DreamTrack-L384 instead of the strongest listed baseline | visual object tracking · NFS | SUC: 67.6 -> 67.4 | Hurt |
| add Future Dreaming Module (ablation) | DreamTrack baseline · LaSOT | SUC: 69.3 -> 71.7 | Helped |
| add Temporal Self-Attention (ablation) | DreamTrack with Future Dreaming Module · LaSOT | SUC: 71.7 -> 72.6 | Helped |
| add Temporal Cross-Attention (ablation) | DreamTrack with Future Dreaming and Temporal Self-Attention · LaSOT | SUC: 72.6 -> 73.1 | Helped |
| add Multimodal Prediction (ablation) | DreamTrack with Temporal Self/Cross-Attention · LaSOT | SUC: +0.7% SUC | Helped |
| add Multimodal Prediction (ablation) | DreamTrack with Temporal Self/Cross-Attention · TNL2K | SUC: +0.9% SUC | Helped |
| remove dreaming and reconstruction losses (ablation) | DreamTrack256 · LaSOT | SUC: 73.8 -> 68.3 | Hurt |
Surprising findings
- Predicted future frames (t+1, t+2) are notably less accurate than current-frame predictions, with SUC dropping to 68.5% at two steps ahead on LaSOT in the evaluation table.
Limitations
- Future predictions degrade notably with horizon (two-step-ahead SUC drops to 68.5% on LaSOT).
- The largest variant is slow (32 FPS, 359 GFLOPs), and the high-cost configuration is required for top results.
Experimental setup
Datasets: COCO LaSOT TrackingNet GOT-10k
Benchmarks: LaSOT LaSOText GOT-10k TrackingNet TNL2K OTB100 NFS UAV123
Metrics: SUC PNorm P AO SR0.5 SR0.75 FPS FLOPs Params
Baselines: OSTrack LoRAT ARTrackV2 SeqTrack ARTrack MixFormer STARK SwinTrack TransT ROMTrack
Builds on: OSTrack Dreamer MixFormer SeqTrack ARTrack
Closest papers by meaning
- VastTrack: Vast Category Visual Object Tracking (NeurIPS 2024 · similarity 0.87)
- Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking (AAAI 2025 · similarity 0.87)
- HIPTrack: Visual Tracking with Historical Prompts (CVPR 2024 · similarity 0.86)
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.86)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.86)
- Explicit Visual Prompts for Visual Object Tracking (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.