Research map / Video / Video generation
TrackGo: A Flexible and Efficient Method for Controllable Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TrackGo, a motion-controllable image-to-video method that uses free-form masks and arrows converted to point trajectories, injected into SVD's temporal self-attention via a lightweight TrackAdapter with an attention loss.
Problem
Precise, efficient control of object and part motion in diffusion-based controllable video generation, where prior box-, point-, or ControlNet-based conditioning is imprecise and parameter-heavy.
Approach
Free-form masks plus arrows define the motion region and trajectory; the resulting point trajectories are injected by a parallel TrackAdapter branch in each temporal self-attention layer of SVD, with a mask suppressing the original branch on the motion region and an attention loss that accelerates convergence.
Key results
- On VIPSeg, TrackGo reports FVD 248.27, FID 25.60, ObjMC 191.15, versus DragAnything 294.91/28.16/236.02 and DragNUWA 321.31/30.15/298.98.
- On the internal validation set, TrackGo reports FVD 136.11 and ObjMC 79.52 against DragAnything 169.73/133.89 and DragNUWA 178.37/129.80.
- TrackGo adds 29.36M parameters and runs in 33.94s per video, versus 685.06M and 152.98s for DragAnything; 62% of user study votes went to TrackGo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TrackGo rather than DragAnything | SVD-based controllable video generation · VIPSeg validation | FVD: 294.91 -> 248.27 | Helped |
| Use TrackGo rather than DragAnything | SVD-based controllable video generation · VIPSeg validation | FID: 28.16 -> 25.60 | Helped |
| Use TrackGo rather than DragAnything | SVD-based controllable video generation · VIPSeg validation | ObjMC: 236.02 -> 191.15 | Helped |
| Use TrackGo rather than DragAnything | SVD-based controllable video generation · Internal validation dataset | FVD: 169.73 -> 136.11 | Helped |
| Use TrackGo rather than DragAnything | SVD-based controllable video generation · Internal validation dataset | FID: 32.85 -> 29.19 | Helped |
| Use TrackGo rather than DragNUWA | SVD-based controllable video generation · Internal validation dataset | ObjMC: 129.80 -> 79.52 | Helped |
| Add attention loss (ablation) | TrackGo · Internal validation dataset, across 14k, 16k, and 18k steps | FVD: 216.54 -> 204.02; 191.54 -> 184.03; 165.12 -> 136.11 | Helped |
| Add attention mask and attention loss (ablation) | TrackGo · Internal validation dataset, across 14k, 16k, and 18k steps | FVD: 219.15 -> 204.02; 208.31 -> 184.03; 218.50 -> 136.11 | Helped |
Surprising findings
- Attention maps of SVD's temporal self-attention layers already highlight moving regions, which motivates injecting control there.
- Setting the unspecified-area intensity tau to 0 gave more cohesive videos than strong suppression, so suppression is not always desirable.
Limitations
- Struggles with large movements and some detail distortion (e.g., bird head in late frames).
- Cannot handle 3D rotation; often interprets rotation as translation.
- Trained on an internal proprietary dataset, limiting reproducibility.
Experimental setup
Datasets: internal video dataset (~110K filtered clips) VIPSeg
Benchmarks: VIPSeg internal validation set (300 videos)
Metrics: FVD FID ObjMC parameters inference time user study votes
Baselines: DragNUWA DragAnything
Builds on: Stable Video Diffusion Animate Anyone Co-Tracker DEVA
Closest papers by meaning
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.88)
- Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation (CVPR 2025 · similarity 0.88)
- PEEKABOO: Interactive Video Generation via Masked-Diffusion (CVPR 2024 · similarity 0.88)
- IM-Zero: Instance-level Motion Controllable Video Generation in a Zero-shot Manner (CVPR 2025 · similarity 0.88)
- ControlVideo: Training-free Controllable Text-to-video Generation (ICLR 2024 · similarity 0.87)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.