Research map / Video / Video generation
Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Track4Gen fine-tunes Stable Video Diffusion with an added point-tracking loss on refined U-Net features, reducing appearance drift in image-to-video generation while enabling point tracking.
Problem
Video diffusion generators suffer from appearance drift (objects gradually degrade or change across frames) because training supervision is only the pixel/latent denoising loss with no explicit feature-level spatial tracking supervision.
Approach
Jointly train a video diffusion model with the denoising loss and a dense point-correspondence loss computed on features from an intermediate decoder block passed through a trainable identity-initialized refiner, with the refined features fed back to the backbone via a zero convolution with stop-gradient.
Key results
- Track4Gen reaches Subject Consistency 0.9746 vs 0.9535 for pre-trained SVD on VBench-I2V and the lowest FID (26.6 vs 29.0).
- On zero-shot DAVIS-480p (24-frame) tracking, Track4Gen features reach δx_avg 69.7 vs 42.4 for SVD, approaching RAFT (73.3).
- With DINO-Tracker test-time adaptation, Track4Gen features reach a DAVIS AJ of 55.7 vs 64.6 for DINO-Tracker with DINOv2 features.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add point-correspondence supervision with the refiner module | Stable Video Diffusion · VBench-I2V | Subject Consistency: 0.9665 -> 0.9746 | Helped |
| add point-correspondence supervision with the refiner module | Stable Video Diffusion · DAVIS | FID: 27.0 -> 26.6 | Helped |
| add point-correspondence supervision with the refiner module | Stable Video Diffusion · DAVIS | FVD: 735 -> 724 | Helped |
| remove the refiner module (ablation) | Track4Gen · VBench-I2V | Subject Consistency: 0.9746 -> 0.9506 | Hurt |
| train spatial and temporal transformers instead of temporal transformers only (ablation) | Track4Gen · VBench-I2V | Subject Consistency: 0.9746 -> 0.9734 | No clear effect |
| train spatial transformers only instead of temporal transformers only (ablation) | Track4Gen · VBench-I2V | Subject Consistency: 0.9746 -> 0.9726 | No clear effect |
| replace 2D refiner convolutions with 3D convolutions (ablation) | Track4Gen · VBench-I2V | Subject Consistency: 0.9746 -> 0.9687 | Hurt |
| add synthetic videos to real-video training data (ablation) | Track4Gen · VBench-I2V | Subject Consistency: 0.9747 -> 0.9708 | Hurt |
| add synthetic videos to real-video training data (ablation) | Track4Gen · BADJA | δseg: 40.4 -> 42.1 | Helped |
| add synthetic videos to real-video training data (ablation) | Track4Gen · BADJA | δ3px: 5.0 -> 4.8 | Hurt |
Surprising findings
- Adding synthetic Kubric videos with tracks degraded video generation quality, although tracks from real videos and synthetic videos gave similar tracking guidance.
Limitations
- Generated videos tend to show less dynamic motion than other video generators.
- Training set is small (567 video-trajectory pairs) and trajectories come from optical flow chaining rather than ground-truth annotations.
Experimental setup
Datasets: DAVIS BADJA TAP-Vid DAVIS VBench-I2V Kubric MOVi-E
Benchmarks: VBench-I2V DAVIS TAP-Vid DAVIS BADJA
Metrics: Subject Consistency Temporal Flickering Motion Smoothness Imaging Quality Video-Image Alignment FID FVD position accuracy occlusion accuracy average Jaccard
Baselines: SVD finetuned SVD ZeroScope RAFT DINO-Tracker TAPIR PIPs++ Omnimotion TAP-Net
Builds on: Stable Video Diffusion ControlNet DINO-Tracker EDM
Closest papers by meaning
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.90)
- Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision (NeurIPS 2025 · similarity 0.89)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.89)
- DiffusionTrack: Point Set Diffusion Model for Visual Object Tracking (CVPR 2024 · similarity 0.89)
- Track-On: Transformer-based Online Point Tracking with Memory (ICLR 2025 · similarity 0.89)
- Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.