Research map / Video / Video generation
Follow-Your-Click: Open-domain Regional Image Animation via Motion Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RegionMaker, a click-plus-short-motion-prompt regional image animation method built on SD1.5 and AnimateDiff, using SAM masks, first-frame masking, a motion-augmented cross-attention module, and optical-flow motion strength control.
Problem
Image-to-video generation lacks regional control and requires long, redundant text prompts; users want to animate only selected objects with simple interaction and short motion cues.
Approach
Combine SAM-derived click masks, optical-flow-based motion masks as training conditions, random masking of the first-frame latent during training, a motion-augmented cross-attention module trained on a GPT-4-re-annotated short-motion caption set (WebVid-Motion), and flow-magnitude embeddings for motion strength.
Key results
- Ours achieves FVD 271.74 and I1-MSE 21.46 versus Pika Labs 337.84 and 33.27 on the 30-prompt benchmark.
- Ours reaches text alignment 0.7981 and temporal consistency 0.9613, with Pika Labs at 0.7163 and 0.9724 respectively.
- Ablation: removing data and motion-augmented module raises FVD from 271.74 to 698.21 and lowers text alignment from 0.7981 to 0.3659.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RegionMaker instead of the strongest reported baseline | Regional image animation · Regional image animation benchmark | Text-Align: 0.7163 -> 0.7981 | Helped |
| use RegionMaker instead of the strongest reported baseline | Regional image animation · MSRVTT | FVD: 337.84 -> 271.74 | Helped |
| add WebVid-Motion data and the motion-augmented module (ablation) | RegionMaker · Regional image animation benchmark | Text-Align: 0.3659 -> 0.7981 | Helped |
| add WebVid-Motion data and the motion-augmented module (ablation) | RegionMaker · MSRVTT | FVD: 698.21 -> 271.74 | Helped |
| add the motion-augmented module (ablation) | RegionMaker · Regional image animation benchmark | Text-Align: 0.6294 -> 0.7981 | Helped |
| add the motion-augmented module (ablation) | RegionMaker · MSRVTT | FVD: 391.47 -> 271.74 | Helped |
| add WebVid-Motion data (ablation) | RegionMaker · Regional image animation benchmark | Text-Align: 0.4356 -> 0.7981 | Helped |
| add WebVid-Motion data (ablation) | RegionMaker · MSRVTT | FVD: 562.33 -> 271.74 | Helped |
| randomly mask the first-frame image latent during training (mask ratio 0.7) (ablation) | RegionMaker · UCF-101 and MSRVTT | FVD | Helped |
| use optical-flow magnitude rather than FPS for motion-strength control (ablation) | RegionMaker · Regional image animation | motion strength control | Helped |
Surprising findings
- The optimal first-frame mask ratio is surprisingly high (70%), and training without masking gives poor motion and temporal consistency.
Limitations
- Struggles with large and complex human motions due to limited training samples.
- Relies on commercial and closed baselines accessed at a specific date, so results may not be reproducible.
- Quantitative benchmark is small (30 prompts) and user study uses 32 subjects.
Experimental setup
Datasets: WebVid-10M WebVid-Motion MSRVTT UCF-101
Benchmarks: Pixabay-based 30-prompt benchmark MSRVTT UCF-101
Metrics: I1-MSE Temporal consistency (CLIP) Text alignment (CLIP) FVD Perceptual Input Conformity user study rankings
Baselines: Gen-2 Genmo Pika Labs Dynamicrafter I2VGen-XL SVD Animate-anything
Builds on: Latent Diffusion Models AnimateDiff Stable Diffusion v1.5 SAM WebVid-10M
Closest papers by meaning
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.90)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.88)
- Frame In-N-Out: Unbounded Controllable Image-to-Video Generation (NeurIPS 2025 · similarity 0.88)
- Motion Prompting: Controlling Video Generation with Motion Trajectories (CVPR 2025 · similarity 0.88)
- Framer: Interactive Frame Interpolation (ICLR 2025 · similarity 0.88)
- TokenMotion: Decoupled Motion Control via Token Disentanglement for Human-centric Video Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.