Research map / Video / Video generation
Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a pretrained image-to-video diffusion model (SVD) for keyframe interpolation by lightly fine-tuning a backward-motion variant via 180-degree rotated temporal attention maps and fusing forward and backward denoising predictions during sampling.
Problem
Generating coherent in-between video between two distant keyframes (>=1s apart) with large motion, which existing frame interpolation methods and most open image-to-video models handle poorly without costly training.
Approach
Reverse the arrow of time in a pretrained image-to-video model by rotating its temporal self-attention maps 180 degrees and fine-tuning only the value and output projections (W_v, W_o) to produce backward-moving videos from the end keyframe; then fuse forward and backward noise predictions at every diffusion sampling step for forward-backward motion consistency.
Key results
- On Pexels, the method achieves FID 22.99 and FVD 306.84, versus FILM 25.16/371.83, TRF 31.43/563.16 and DynamiCrafter 32.06/393.12.
- On Davis, the method achieves FID 32.68 and FVD 424.69, versus FILM 41.85/1048.65, TRF 36.79/563.07 and DynamiCrafter 38.32/439.74.
- Fine-tuning fewer than 2% of U-Net parameters with about 100 training videos suffices to outperform the baselines; 50 training videos still beat the baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dual-directional sampling with lightweight backward-motion fine-tuning | Stable Video Diffusion · Pexels | FID: 25.16 -> 22.99 | Helped |
| Use dual-directional sampling with lightweight backward-motion fine-tuning | Stable Video Diffusion · Pexels | FVD: 371.83 -> 306.84 | Helped |
| Use dual-directional sampling with lightweight backward-motion fine-tuning | Stable Video Diffusion · Davis | FID: 36.79 -> 32.68 | Helped |
| Use dual-directional sampling with lightweight backward-motion fine-tuning | Stable Video Diffusion · Davis | FVD: 439.74 -> 424.69 | Helped |
| Add rotated temporal attention maps and fine-tune Wv, Wo instead of fine-tuning all temporal attention projections without the map (ablation) | Ours · Pexels | FID: 26.42 -> 22.99 | Helped |
| Add rotated temporal attention maps and fine-tune Wv, Wo instead of fine-tuning all temporal attention projections without the map (ablation) | Ours · Davis | FVD: 549.98 -> 424.69 | Helped |
| Fine-tune Wv, Wo for backward motion (ablation) | Ours · Pexels | FID: 37.68 -> 22.99 | Helped |
| Fine-tune Wv, Wo for backward motion (ablation) | Ours · Davis | FVD: 604.76 -> 424.69 | Helped |
| Set motion bucket id to 65 instead of 127 (ablation) | Ours · Pexels | FVD: 306.84 -> 270.27 | Helped |
| Set motion bucket id to 65 instead of 127 (ablation) | Ours · Davis | FVD: 424.69 -> 491.85 | Hurt |
| Set motion bucket id to 255 instead of 127 (ablation) | Ours · Pexels | FID: 22.99 -> 22.18 | Helped |
| Set motion bucket id to 255 instead of 127 (ablation) | Ours · Davis | FID: 32.68 -> 34.06 | Hurt |
Surprising findings
- Fine-tuning all temporal attention parameters without the rotated attention map gives worse results than the lightweight W_v,o variant, showing that the rotated-map conditioning is what keeps the two motion paths consistent.
- Naively reversing the output of a forward image-to-video model fails because real-world motion is asymmetric, motivating the fine-tuning step.
Limitations
- Inherits motion quality limits of SVD, which struggles with articulated, non-rigid motion such as limbs.
- Fails when keyframes have sparse correspondences, causing artifacts in intermediate frames.
- SVD's strong input-driven motion priors may not represent the motion needed to connect distant keyframes.
Experimental setup
Datasets: 100 synthetic videos generated by SVD from a community website
Benchmarks: Davis Pexels
Metrics: FID FVD
Baselines: FILM TRF DynamiCrafter
Builds on: Stable Video Diffusion MultiDiffusion FILM TRF
Closest papers by meaning
- ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler (ICLR 2025 · similarity 0.91)
- EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation (CVPR 2025 · similarity 0.91)
- Generative Inbetweening through Frame-wise Conditions-Driven Video Generation (CVPR 2025 · similarity 0.90)
- Video Interpolation with Diffusion Models (CVPR 2024 · similarity 0.90)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.90)
- Efficient Video Diffusion Models via Content-Frame Motion-Latent Decomposition (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.