Research map / Video / Video generation
Emergent Temporal Correspondences from Video Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DiffTrack, a framework that analyzes how video DiTs (CogVideoX, HunyuanVideo) establish temporal correspondences via query-key attention, and applies the findings to zero-shot point tracking and a training-free Cross-Attention Guidance for motion-enhanced generation.
Problem
How video Diffusion Transformers internally establish and represent temporal correspondences across frames during generation, and how these correspondences can be explicitly extracted and used.
Approach
Builds a pseudo-ground-truth tracking dataset from prompt-generated videos (CoTracker annotations) and proposes three metrics (matching accuracy, confidence score, attention score) to analyze full 3D attention layer-by-layer and timestep-by-timestep; then uses the identified dominant layer/timestep for zero-shot tracking and zeroes cross-frame attention in dominant layers as a negative guidance te
Key results
- DiffTrack on CogVideoX-5B achieves 46.9 average accuracy on TAP-Vid-DAVIS and 49.2 on Kinetics, outperforming DINOv2 (42.1 / 38.4) and other vision foundation and self-supervised baselines.
- CAG on CogVideoX-2B raises human 2AFC video quality preference from 12.85% to 87.16% and improves VBench dynamic degree from 0.7917 to 0.8235.
- Query-key matching consistently outperforms intermediate feature matching, and temporal matching concentrates in a few layers (e.g., layer 17 in CogVideoX-2B) at the final denoising timestep t=1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use query-key matching instead of intermediate feature matching | DiffTrack temporal correspondence · CogVideoX-2B generated videos | matching accuracy | Helped |
| Use query-key matching instead of intermediate feature matching | DiffTrack temporal correspondence · CogVideoX-5B, HunyuanVideo, and CogVideoX-2B-I2V | matching accuracy | Helped |
| Concentrate temporal matching on a few dominant layers | DiffTrack temporal correspondence · CogVideoX-2B generated videos | harmonic mean of matching metrics | Helped |
| Progress denoising toward later timesteps | DiffTrack temporal correspondence · CogVideoX-2B generated videos | matching accuracy | Mixed |
| Add Cross-Attention Guidance | CogVideoX-2B · Video generation | Dynamic Degree: 0.7917 -> 0.8235 | Helped |
| Add Cross-Attention Guidance | CogVideoX-5B · Video generation | Dynamic Degree: 0.6667 -> 0.6863 | Helped |
| Add Cross-Attention Guidance | CogVideoX-2B and CogVideoX-5B · Video generation human evaluation | Video Quality votes: 12.85 -> 87.16; 39.10 -> 60.90 | Helped |
| Remove temporal compression (set q=1) (ablation) | DiffTrack zero-shot point tracking · TAP-Vid-DAVIS | δx_avg: 26.9 -> 32.3 | Helped |
| Insert the global first frame into every chunk (ablation) | DiffTrack zero-shot point tracking · TAP-Vid-DAVIS | δx_avg: 32.3 -> 44.6 | Helped |
| Interleave subsequent frames within each chunk (ablation) | DiffTrack zero-shot point tracking · TAP-Vid-DAVIS | δx_avg: 44.6 -> 46.3 | Helped |
| Use layer 17 at timestep 1 rather than averaging over timesteps or layers (ablation) | DiffTrack zero-shot point tracking · Object dataset | δ3 matching accuracy: 15.34 -> 63.50; 16.69 -> 63.50 | Helped |
Surprising findings
- Denoising-time temporal matching is best at the final timestep (t=1), meaning inversion error is negligible for the tracking application.
- CogVideoX-2B-I2V performs far worse on DAVIS (16.4 average accuracy) because image-to-video finetuning preserves the first frame and loses dynamic matching capability.
Limitations
- Analysis relies on pseudo ground-truth tracks from CoTracker rather than human-annotated tracks for generated videos.
- CogVideoX-2B-I2V shows substantially degraded temporal matching, so the framework's findings do not transfer uniformly across video DiT variants.
- Human evaluation uses a limited set of 25 participants with 750 responses per baseline.
Experimental setup
Datasets: VBench prompts DAVIS TAP-Vid-DAVIS TAP-Vid-Kinetics
Benchmarks: TAP-Vid-DAVIS TAP-Vid-Kinetics VBench
Metrics: PCK (delta thresholds, average accuracy) matching accuracy confidence score attention score subject consistency background consistency dynamic degree imaging quality 2AFC human preference
Baselines: DINO DINOv2 DINOv2-Reg DIFT (SD1.5) DIFT (SD2.1) SMTC CRW Spa-then-Temp VFS SVD ZeroCo
Builds on: Diffusion Transformers PAG (Perturbed-Attention Guidance) DIFT CoTracker DDIM inversion
Closest papers by meaning
- Video Motion Transfer with Diffusion Transformers (CVPR 2025 · similarity 0.90)
- Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation (CVPR 2025 · similarity 0.90)
- TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking (NeurIPS 2026 · similarity 0.90)
- Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision (NeurIPS 2025 · similarity 0.90)
- DINTR: Tracking via Diffusion-based Interpolation (NeurIPS 2024 · similarity 0.89)
- FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.