Research map / Video / Video generation
Framer: Interactive Frame Interpolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Framer fine-tunes Stable Video Diffusion for keypoint-trajectory-controlled interactive frame interpolation between two images, with an autopilot mode that estimates trajectories via bi-directional feature matching.
Problem
Frame interpolation between two distant or stylistically different images is ambiguous and deterministic VFI methods fail on large motion and appearance change; users lack control over the transition.
Approach
Adapt an image-to-video diffusion model (SVD) to take an end frame as an extra condition, then add a ControlNet-style branch conditioned on Gaussian heatmaps of user-specified or automatically estimated point trajectories, with autopilot trajectories refined during denoising by bi-directional diffusion-feature matching.
Key results
- On DAVIS-7, Framer achieves FVD 115.65 versus 162.86 for the best baseline (FILM), and on UCF101-7 FVD 181.55 versus 239.48 for FILM.
- With Co-Tracker trajectories, Framer reaches FVD 102.31 on DAVIS-7 and 159.87 on UCF101-7.
- In a user study with 20 participants and 1,000 ratings, raters strongly preferred Framer outputs over existing methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Framer autopilot instead of FILM | Framer · DAVIS-7 | FVD: 162.86 -> 115.65 | Helped |
| Use Framer autopilot instead of FILM | Framer · UCF101-7 | FVD: 239.48 -> 181.55 | Helped |
| Add trajectory guidance (ablation) | Framer · DAVIS-7 | FVD: 128.71 -> 115.65 | Helped |
| Add trajectory guidance (ablation) | Framer · UCF101-7 | FVD: 195.54 -> 181.55 | Helped |
| Add trajectory updating (ablation) | Framer · DAVIS-7 | FVD: 120.73 -> 115.65 | Helped |
| Add trajectory updating (ablation) | Framer · UCF101-7 | FVD: 187.37 -> 181.55 | Helped |
| Add bidirectional consistency verification during trajectory updating (ablation) | Framer · DAVIS-7 | FVD: 116.81 -> 115.65 | Helped |
| Add bidirectional consistency verification during trajectory updating (ablation) | Framer · UCF101-7 | FVD: 183.74 -> 181.55 | Helped |
| Use 5 ground-truth point trajectories estimated with Co-Tracker | Framer · DAVIS-7 | FVD: 115.65 -> 102.31 | Helped |
| Use 5 ground-truth point trajectories estimated with Co-Tracker | Framer · UCF101-7 | FVD: 181.55 -> 159.87 | Helped |
Surprising findings
- Reconstruction metrics such as PSNR, SSIM and LPIPS are argued to be unreliable for interpolation, so the paper emphasizes FVD and user preference.
- Early diffusion steps matter more than late steps for correspondence guidance.
Limitations
- Inherits limitations of the pre-trained video diffusion model.
- Fails when no matched points can be found between very different start and end frames.
- Supports only drag control; other interaction types are not explored and transitions between different clips are challenging.
Experimental setup
Datasets: OpenVidHD-0.4M DAVIS UCF101
Benchmarks: DAVIS-7 UCF101-7
Metrics: PSNR SSIM LPIPS FID FVD user preference
Baselines: AMT RIFE FLAVR FILM LDMVFI DynamicCrafter SVDKFI
Builds on: Stable Video Diffusion ControlNet DragNUWA DragAnything DragDiffusion
Closest papers by meaning
- Follow-Your-Click: Open-domain Regional Image Animation via Motion Prompts (AAAI 2025 · similarity 0.88)
- Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation (ICLR 2025 · similarity 0.87)
- Generative Inbetweening through Frame-wise Conditions-Driven Video Generation (CVPR 2025 · similarity 0.87)
- Frame In-N-Out: Unbounded Controllable Image-to-Video Generation (NeurIPS 2025 · similarity 0.86)
- EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation (CVPR 2025 · similarity 0.86)
- ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.