Research map / Video / Video generation
Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an Extrapolating and Decoupling framework for image-to-video diffusion that adds a text-conditioned adapter to DynamiCrafter's temporal attention, extrapolates weights to amplify motion degree, and decouples motion parameters injected over denoising steps.
Problem
Image-to-video diffusion models either produce videos with limited motion degree or motion that fails to follow the text condition, while preserving the input image's appearance.
Approach
Applies model merging to I2V: a learnable Q-Former adapter injects text into temporal attention and is fine-tuned; a training-free weight extrapolation (reversing fine-tuning) restores motion degree; DARE-pruned parameter groups for control, degree and consistency are merged via task arithmetic and switched across denoising steps.</1ntml:parameter>
Key results
- On VBench I2V, the method reaches Avg. 84.25 vs 79.93 for DynamiCrafter, with motion degree 87.64 vs 68.54 and motion control 43.87 vs 30.85.
- On UCF101, motion degree is 77.93 vs 68.17 for DynamiCrafter, and IS improves to 13.71 from 13.31.
- Ablation shows extrapolation alone raises motion degree to 93.33 but reduces motion control and consistency; combining with decoupling gives the best overall balance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply the full Extrapolating and Decoupling framework | DynamiCrafter · VBench I2V | Motion Degree: 68.54 -> 87.64 | Helped |
| Apply the full Extrapolating and Decoupling framework | DynamiCrafter · VBench I2V | Motion Control: 30.85 -> 43.87 | Helped |
| Apply the full Extrapolating and Decoupling framework | DynamiCrafter · UCF101 | FVD: 288.01 -> 324.79 | Hurt |
| Apply the full Extrapolating and Decoupling framework | DynamiCrafter · UCF101 | IS: 13.31 -> 13.71 | Helped |
| Fine-tune with motion-controllability adaptation (ablation) | DynamiCrafter · VBench I2V | Motion Control: 22.85 -> 49.31 | Helped |
| Add extrapolation to adaptation (ablation) | DynamiCrafter · VBench I2V | Motion Degree: 22.85 -> 71.68 | Helped |
| Add decoupling to adaptation and extrapolation (ablation) | DynamiCrafter · VBench I2V | Motion Control: 26.71 -> 43.87 | Helped |
| Add decoupling to adaptation and extrapolation (ablation) | DynamiCrafter · VBench I2V | Motion Degree: 71.68 -> 87.64 | Helped |
| Add adaptation to extrapolation and decoupling (ablation) | DynamiCrafter · VBench I2V | Motion Control: 21.85 -> 43.87 | Helped |
| Use extrapolation without decoupling (ablation) | DynamiCrafter · VBench I2V | I2V Subject Consistency: 97.03 -> 91.11 | Hurt |
| Apply extrapolation without decoupling (ablation) | DynamiCrafter · VBench I2V | Motion Control: 49.31 -> 26.71 | Hurt |
Surprising findings
- Naive fine-tuning of DynamiCrafter on WebVid-2M sharply drops motion degree (68.54 to 11.67), and extrapolating back toward the pre-trained weights restores and amplifies motion.
Limitations
- Extrapolation strength alpha trades motion degree against video quality and consistency.
- Extrapolation alone unlearns camera-motion ability and degrades motion control and consistency.
Experimental setup
Datasets: WebVid-2M UCF101
Benchmarks: VBench I2V UCF101
Metrics: I2V Subject Consistency I2V Background Consistency video quality motion degree motion control FVD Inception Score
Baselines: ConsistI2V SEINE Animate-Anything VideoCrafter SVD-XT I2VGen-XL DynamiCrafter CIL
Builds on: DynamiCrafter Task Arithmetic DARE Conditional Image Leakage
Closest papers by meaning
- Enhancing Motion in Text-to-Video Generation with Decomposed Encoding and Conditioning (NeurIPS 2024 · similarity 0.91)
- Decouple Content and Motion for Conditional Image-to-Video Generation (AAAI 2024 · similarity 0.90)
- MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generation (CVPR 2025 · similarity 0.90)
- Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model (NeurIPS 2024 · similarity 0.90)
- Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation (ICLR 2025 · similarity 0.89)
- SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.