Research map / Detection and segmentation / Tracking
Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that pretrained video diffusion models (I2VGen-XL) encode motion at high noise levels and fuses these features with image diffusion (ADM) appearance features to build TED, a self-supervised label-propagation tracker that better separates visually similar objects.
Problem
Self-supervised video object tracking via label propagation fails when visually similar objects must be distinguished, because existing methods rely on appearance features and process frames independently.
Approach
Extract motion-aware features from the internal activations of a pretrained 3D video diffusion UNet at high noise levels, where motion is isolated, and fuse them with per-frame image diffusion appearance features for training-free label propagation.
Key results
- On DAVIS-2017, TED reaches 77.6 J&F_m, up to 6 points above recent self-supervised methods such as SFC (71.2) and Spa-then-Temp (74.1).
- On Kubric-Similar with identical balls, TED achieves J_m of 87.2 while most baselines stay near 50.
- On Youtube-Similar, TED reaches 66.0 J&F_m, a gain of 6.4 over Spa-then-Temp and 5.3 over DIFT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TED's video-diffusion motion features and fuse them with appearance features | self-supervised video label propagation · DAVIS-2017 val | J&F_m: 74.1 -> 77.6 | Helped |
| add TED's motion-aware features for similar-looking objects | self-supervised video label propagation · Youtube-Similar | J&F_m: 60.7 -> 66.0 | Helped |
| add TED's motion-aware features for identical-looking objects | self-supervised video label propagation · Kubric-Similar | J_m: 64.7 -> 87.2 | Helped |
| combine motion and appearance features instead of using motion features alone (ablation) | TED · DAVIS and Youtube-Similar | tracking accuracy | Helped |
| use motion features alone (lambda=1.0) rather than a motion-appearance mixture (ablation) | TED · Kubric-Similar | tracking accuracy | Helped |
| extract motion features at high noise levels (ablation) | TED motion representation · Youtube-Similar and Kubric-Similar | tracking accuracy | Helped |
| extract motion features from block 3 rather than blocks 1, 2, or 4 (ablation) | TED motion representation · DAVIS | J&F_m: 24.8, 47.6, 31.5 -> 66.3 | Helped |
| replace I2VGen-XL with SVD for motion features, retaining ADM appearance features (ablation) | TED · DAVIS | J&F_m: 77.6 -> 76.6 | Hurt |
| replace ADM with SD for appearance features, retaining I2VGen motion features (ablation) | TED · DAVIS | J&F_m: 77.6 -> 71.7 | Hurt |
| replace both default diffusion models with SVD + SD (ablation) | TED · DAVIS | J&F_m: 77.6 -> 71.5 | Hurt |
| add overlapping frames between video clips instead of using no overlap (ablation) | TED · Youtube-Similar | tracking accuracy | Helped |
| remove averaging representations computed from multiple noisy inputs (ablation) | TED · DAVIS | J&F_m: 77.6 -> 77.2 | Hurt |
Surprising findings
- Motion features from video diffusion models peak at high noise levels (tau=600-900), while appearance features degrade there.
- Many self-supervised baselines score near 50% J_m on identical-object Kubric-Similar, equivalent to random guessing.
Limitations
- Higher computation time and memory than prior self-supervised methods such as SFC and DIFT.
- Relies on fixed, empirically chosen diffusion timesteps and fusion weights per dataset.
- Evaluated only on label propagation style tasks; the Youtube-Similar and Kubric-Similar benchmarks are small (28 and 30 videos).
Experimental setup
Datasets: DAVIS-2017 YouTube-VOS Kubric
Benchmarks: DAVIS-2017 Youtube-Similar Kubric-Similar JHMDB VIP YouTube-VOS MOSE
Metrics: J&F_m J_m F_m PCK@0.1 PCK@0.2 mIoU
Baselines: DIFT SFC Spa-then-Temp SMTC CRW VFS UVC MoCo SimCLR BYOL SimSiam InstDis
Builds on: DIFT Spa-then-Temp CRW Two-Stream ConvNets I2VGen-XL
Closest papers by meaning
- DiffusionTrack: Diffusion Model for Multi-Object Tracking (AAAI 2024 · similarity 0.91)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.90)
- DiffusionTrack: Point Set Diffusion Model for Visual Object Tracking (CVPR 2024 · similarity 0.90)
- DINTR: Tracking via Diffusion-based Interpolation (NeurIPS 2024 · similarity 0.89)
- Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation (CVPR 2025 · similarity 0.89)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.