Research map / Video / Video generation
SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SG-I2V, a zero-shot, self-guided trajectory control method for image-to-video diffusion (SVD) that aligns self-attention features across frames and optimizes the latent within bounding-box trajectories, with high-frequency-preserving post-processing.
Problem
Controlling object motion and camera movement in image-to-video generation typically requires fine-tuning on annotated motion data, which is costly and data-hungry; existing zero-shot trajectory control methods target text-to-video and cannot animate existing images.
Approach
Modify spatial self-attention so every frame attends to the first frame's keys and values (stop-gradient) to obtain cross-frame semantically aligned features, then optimize the noisy latent in early denoising steps so features inside moving bounding boxes follow the trajectory, and restore high-frequency components of the original latent to avoid quality loss.
Key results
- SG-I2V achieves FID 28.87, FVD 298.10, ObjMC 14.43 on VIPSeg, outperforming zero-shot baselines (e.g., DragDiffusion† ObjMC 31.49, SVD no control ObjMC 39.59) across all metrics.
- Versus supervised SVD-based DragNUWA v1.5 (ObjMC 10.84) and DragAnything (ObjMC 11.64), SG-I2V has slightly worse motion fidelity but comparable or better FID (28.87 vs 30.73/30.81) and at full 576x1024 resolution.
- Average runtime of 305 seconds per video on an A6000 48GB, with about 30 GB peak GPU memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SG-I2V instead of the strongest zero-shot baseline on FID, DragDiffusion† | SVD image-to-video generation · VIPSeg validation | FID: 30.93 -> 28.87 | Helped |
| use SG-I2V instead of the strongest zero-shot baseline on FVD, MOFT† | SVD image-to-video generation · VIPSeg validation | FVD: 402.09 -> 298.10 | Helped |
| use SG-I2V instead of the strongest zero-shot baseline on ObjMC, SVD (No Control) | SVD image-to-video generation · VIPSeg validation | ObjMC: 39.59 -> 14.43 | Helped |
| use SG-I2V instead of DragNUWA v1.5 | SVD image-to-video generation · VIPSeg validation | FID: 30.73 -> 28.87 | Helped |
| use SG-I2V instead of DragNUWA v1.5 | SVD image-to-video generation · VIPSeg validation | ObjMC: 10.84 -> 14.43 | Hurt |
| use Gaussian weighting instead of identity weighting (ablation) | SG-I2V · VIPSeg validation | FID: 28.88 -> 28.87 | Helped |
| use Gaussian weighting instead of identity weighting (ablation) | SG-I2V · VIPSeg validation | FVD: 300.20 -> 298.10 | Helped |
| use Gaussian weighting instead of identity weighting (ablation) | SG-I2V · VIPSeg validation | ObjMC: 14.72 -> 14.43 | Helped |
| replace aligned modified self-attention features with unaligned feature maps (ablation) | SG-I2V · VIPSeg validation | ObjMC | Hurt |
| replace the modified self-attention operation with original self-attention (ablation) | SG-I2V · VIPSeg validation | FVD | Hurt |
| use optimized latent without high-frequency replacement (γ = 1) (ablation) | SG-I2V · VIPSeg validation | FID | Hurt |
| optimize features at later denoising timesteps (ablation) | SG-I2V · VIPSeg validation | FID | Hurt |
Surprising findings
- Upsampling-block features of video diffusion models show only weak cross-frame semantic correspondence, unlike image diffusion models where they are strongly aligned.
- Zero-shot techniques from text-to-video (e.g., FreeTraj noise initialization) do not transfer well to image-to-video models and degrade visual quality.
Limitations
- Base video diffusion model bounds quality for subjects with large motion or complex physical interactions.
- Latent optimization can push latents out of distribution, causing artifacts at higher learning rates.
- Motion fidelity remains slightly below supervised SVD-based methods (ObjMC 14.43 vs 10.84-11.64).
Experimental setup
Datasets: VIPSeg
Benchmarks: VIPSeg
Metrics: FID FVD ObjMC
Baselines: DragNUWA v1.5 DragAnything Image Conductor FreeTraj MOFT DragDiffusion
Builds on: Stable Video Diffusion DragAnything DragDiffusion FreeTraj FreeInit
Closest papers by meaning
- IM-Zero: Instance-level Motion Controllable Video Generation in a Zero-shot Manner (CVPR 2025 · similarity 0.91)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.90)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.89)
- Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think (CVPR 2025 · similarity 0.89)
- Motion-Zero: A Zero-Shot Trajectory Control Framework of Moving Object for Diffusion-Based Video Generation (AAAI 2025 · similarity 0.89)
- I2VControl-Camera: Precise Video Camera Control with Adjustable Motion Strength (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.