Research map / Video / Video generation
IM-Zero: Instance-level Motion Controllable Video Generation in a Zero-shot Manner
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IM-Zero, a training-free two-stage pipeline that generates instance-layout- and trajectory-controlled videos by combining a grounded text-to-image model's keyframes with AnimateDiff-based motion and detail injection.
Problem
Controllable video generation that jointly controls instance spatial layouts and movement trajectories with high video quality and temporal consistency, without costly training.
Approach
Decouples location and motion: an inflated grounded T2I model (InstanceDiffusion) produces keyframe candidates with cross-frame attention to supply motion (Canny/ControlNet) and texture (IP-Adapter) guidance for a coarse video, which is then refined by injecting T2V motion priors and T2I detail priors into the diffusion latents instead of manipulating attention maps.
Key results
- On DAVIS-17, IM-Zero reaches FVD 4070.01 and mIoU 0.27 versus 0.22 for the best baseline, with mIoU improvements of 28.6% and 61.5% over the previous best on the two datasets.
- On GOT10k, IM-Zero achieves FVD 4187.27 and KID 15.92, the best among compared zero-shot methods.
- In T2V generation vs. AnimateDiff, IM-Zero improves Imaging Quality from 0.67 to 0.74 and CLIPSim from 99.05 to 99.73.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use IM-Zero instead of MotionBooth | Instance-level controllable video generation · DAVIS-17 | FVD: 4192.19 -> 4070.01 | Helped |
| Use IM-Zero instead of MotionBooth | Instance-level controllable video generation · DAVIS-17 | mIoU: 0.22 -> 0.27 | Helped |
| Use IM-Zero instead of MotionBooth | Instance-level controllable video generation · GOT10k | FVD: 4311.71 -> 4187.27 | Helped |
| Use IM-Zero instead of MotionBooth | Instance-level controllable video generation · GOT10k | mIoU: 0.14 -> 0.21 | Helped |
| Add first cross-frame attention (CFA) (ablation) | IM-Zero · DAVIS-17 | FVD: 4320.07 -> 4070.01 | Helped |
| Add second cross-frame attention (CFA) (ablation) | IM-Zero · DAVIS-17 | FVD: 4528.09 -> 4070.01 | Helped |
| Use first-keyframe initialization noise (ablation) | IM-Zero · DAVIS-17 | FVD: 4786.17 -> 4070.01 | Helped |
| Add IP-Adapter (ablation) | IM-Zero · DAVIS-17 | FVD: 4549.94 -> 4070.01 | Helped |
| Use IM-Zero instead of AnimateDiff | Text-to-video generation · 50 prompts randomly generated by GPT | Imaging Quality: 0.67 -> 0.74 | Helped |
Surprising findings
- On motion transfer, IM-Zero's Motion Fidelity (63.7) is lower than MotionClone (74.5) and MotionDirector (75.5), though it has the best Imaging Quality.
Limitations
- Video quality and layout alignment are bounded by the pre-trained grounded T2I and T2V models used.
- Temporal consistency is weaker than training-based methods.
- Evaluation uses small subsets (50 DAVIS-17 and 70 GOT10k videos) with automatically extracted boxes.
Experimental setup
Datasets: DAVIS-17 GOT10k
Benchmarks: DAVIS-17 GOT10k MotionClone motion transfer data
Metrics: FVD KID CLIPSim mIoU Centroid Distance Motion Fidelity Imaging Quality
Baselines: T2V-Zero Peekaboo TrailBlazer MotionBooth MOFT MotionClone MotionDirector AnimateDiff
Builds on: InstanceDiffusion AnimateDiff T2V-Zero ControlNet IP-Adapter
Closest papers by meaning
- SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation (ICLR 2025 · similarity 0.91)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.90)
- Motion-Zero: A Zero-Shot Trajectory Control Framework of Moving Object for Diffusion-Based Video Generation (AAAI 2025 · similarity 0.89)
- AnimateAnything: Consistent and Controllable Animation for Video Generation (CVPR 2025 · similarity 0.88)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.88)
- TrackGo: A Flexible and Efficient Method for Controllable Video Generation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.