Research map / Video / Video generation
Generative Inbetweening through Frame-wise Conditions-Driven Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FCVG, which gives each frame an explicit control condition (frame-wise interpolated matched lines plus optional pose) injected into Stable Video Diffusion to make generative inbetweening temporally stable.
Problem
Generative inbetweening from two key frames suffers temporal instability because the interpolation path between them is ambiguous, especially under large motion gaps.
Approach
Extract line matches between start and end frames with GlueStick, linearly interpolate them frame by frame into per-frame control conditions (with DWPose skeletons for humans), and inject these into a pretrained I2V model via ControlNeXt-style lightweight control, combined with time-reversal bidirectional fusion that needs no noise re-injection.
Key results
- At frame gap 23, FCVG reaches FID 24.05 and FVD 437.9, beating GI (31.39 / 524.0) and other generative baselines; FILM has better LPIPS (0.1540 vs 0.1832).
- At frame gap 12, FCVG obtains FVMD 4537.4 vs GI 4721.0 and FID 22.77 vs 32.37.
- Inference uses 25 steps without noise re-injection, giving 523 s versus 975 s for GI at 50 steps on 25x1024x576 videos.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use frame-wise line-matching and pose conditions (ablation) | FCVG · Frame Gap = 23 | FVD: 536.5 -> 437.9 | Helped |
| add frame-wise line-matching and pose conditions (ablation) | FCVG · Frame Gap = 23 | LPIPS: 0.2485 -> 0.1832 | Helped |
| add line-matching conditions (ablation) | FCVG · Frame Gap = 23 | FVD: 498.8 -> 437.9 | Helped |
| add human pose conditions (ablation) | FCVG · Frame Gap = 23 | FVD: 446.1 -> 437.9 | Helped |
| set control weight γ to 0.5 instead of 1.0 (ablation) | FCVG · Frame Gap = 23 | FVD: 437.9 -> 431.4 | Helped |
| set control weight γ to 2.0 instead of 1.0 (ablation) | FCVG · Frame Gap = 23 | FVD: 437.9 -> 441.1 | Hurt |
| use FCVG instead of GI | SVD video inbetweening · Frame Gap = 23 | FVD: 524.0 -> 437.9 | Helped |
| use FCVG instead of GI | SVD video inbetweening · Frame Gap = 12 | FVMD: 4721.0 -> 4537.4 | Helped |
| use FCVG instead of FILM | video inbetweening · Frame Gap = 23 | LPIPS: 0.1540 -> 0.1832 | Hurt |
| use FCVG instead of FILM | video inbetweening · Frame Gap = 23 | FID: 25.00 -> 24.05 | Helped |
| use FCVG instead of FILM | video inbetweening · Frame Gap = 12 | LPIPS: 0.1980 -> 0.2378 | Hurt |
Surprising findings
- Optical-flow method FILM achieves better LPIPS than FCVG, and better FVMD/FVD at small gaps is not claimed by FCVG overall.
Limitations
- Dependence on line matching; incorrect matches arise when input frames are highly similar.
- Sparse matched lines for large differences between input frames reduce effectiveness; control weight adjustment is insufficient.
- Built on SVD, so limited by the underlying I2V model; real-time inference remains infeasible.
Experimental setup
Datasets: DAVIS RealEstate10K Pexels
Benchmarks: in-house test split of 524-clip dataset
Metrics: LPIPS FID FVD VBench FVMD inference time
Baselines: FILM DynamiCrafter TRF GI
Builds on: Stable Video Diffusion ControlNeXt Time reversal (TRF) Generative Inbetweening (GI) GlueStick
Closest papers by meaning
- Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation (ICLR 2025 · similarity 0.90)
- ConditionVideo: Training-Free Condition-Guided Video Generation (AAAI 2024 · similarity 0.89)
- Extrapolating and Decoupling Image-to-Video Generation Models: Motion Modeling is Easier Than You Think (CVPR 2025 · similarity 0.88)
- Decouple Content and Motion for Conditional Image-to-Video Generation (AAAI 2024 · similarity 0.88)
- WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception (NeurIPS 2025 · similarity 0.87)
- Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.