Research map / Image generation / Fast diffusion sampling
Hierarchical Flow Diffusion for Efficient Frame Interpolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes hierarchical flow diffusion that denoises bilateral optical flow coarse-to-fine, conditioned on multiscale encoder features, and feeds the predicted flow to a flow-guided synthesizer trained end to end, for fast and accurate video frame interpolation.
Problem
Diffusion-based video frame interpolation denoises in a large latent space, which is slow, less accurate, and struggles with large motions, while non-diffusion flow-based methods produce oversmoothed results.
Approach
Instead of latent-space denoising, run a diffusion model that denoises bilateral optical flow explicitly in a multi-stage coarse-to-fine hierarchy (one stage per feature level, sharing U-Net weights), conditioned on encoder features, and use the flow to warp features for a flow-guided synthesizer that is jointly fine-tuned.
Key results
- On SNU-FILM extreme, LPIPS 0.0839 and FID 27.032 vs. the best prior diffusion baseline CBBD at 0.1040 and 36.729.
- On Xiph 4K, LPIPS 0.0614 and FID 14.132, beating CBBD (0.0634, 24.621) and SGM-VFI (0.2172, 27.334).
- Inference takes 0.20 s on an RTX-4090 for a 1024x1024 pair, versus 8.3 s for LDMVFI and 2.1 s for CBBD, with about 2.9 GB VRAM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use hierarchical optical-flow diffusion instead of latent-space denoising (ablation) | Hierarchical Flow Diffusion · SNU-FILM hard | FID: 22.283 -> 15.320 | Helped |
| use hierarchical optical-flow diffusion instead of latent-space denoising (ablation) | Hierarchical Flow Diffusion · SNU-FILM extreme | FID: 46.414 -> 27.032 | Helped |
| add explicit intermediate flow representation (ablation) | Hierarchical Flow Diffusion · SNU-FILM hard | FID: 19.674 -> 15.320 | Helped |
| add explicit intermediate flow representation (ablation) | Hierarchical Flow Diffusion · SNU-FILM extreme | FID: 39.642 -> 27.356 | Helped |
| jointly fine-tune the synthesizer and diffusion model (ablation) | Hierarchical Flow Diffusion · SNU-FILM hard | FID: 16.158 -> 15.320 | Helped |
| jointly fine-tune the synthesizer and diffusion model (ablation) | Hierarchical Flow Diffusion · SNU-FILM extreme | FID: 28.767 -> 27.356 | Helped |
| share diffusion-network parameters across levels (ablation) | Hierarchical Flow Diffusion · SNU-FILM hard | FID: 16.324 -> 15.320 | Helped |
| share diffusion-network parameters across levels (ablation) | Hierarchical Flow Diffusion · SNU-FILM extreme | FID: 29.313 -> 27.356 | Helped |
| increase training resolution from 128 ×128 to 256 × 256 (ablation) | Hierarchical Flow Diffusion · SNU-FILM hard | FID: 16.737 -> 15.320 | Helped |
| increase training resolution from 128 ×128 to 512 × 512 (ablation) | Hierarchical Flow Diffusion · SNU-FILM extreme | FID: 29.774 -> 26.925 | Helped |
| increase the number of denoising steps in hierarchical diffusion (ablation) | Hierarchical Flow Diffusion · SNU-FILM hard | FID | Helped |
Surprising findings
- Replacing the flow-based synthesizer's flow with the flow from EMA-VFI and using the proposed synthesizer improved EMA-VFI results, showing the synthesizer alone contributes gains.
- Single-level diffusion variants struggle to converge and gain little from more denoising steps, while the hierarchical version keeps improving.
Limitations
- Artifacts remain in extreme motion cases.
- Training uses relatively small-scale data (Vimeo90k only), which the authors attribute the remaining failures to.
Experimental setup
Datasets: Vimeo90k
Benchmarks: SNU-FILM Xiph DAVIS Vimeo-90k MPI-Sintel
Metrics: LPIPS FID EPE runtime
Baselines: LDMVFI CBBD MADiff SGM-VFI AMT-G EMA-VFI URPNet Per-VFI VFIformer
Builds on: DDPM DDIM RAFT LPIPS
Closest papers by meaning
- EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation (CVPR 2025 · similarity 0.90)
- ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler (ICLR 2025 · similarity 0.89)
- Disentangled Motion Modeling for Video Frame Interpolation (AAAI 2025 · similarity 0.89)
- Video Diffusion Models Are Strong Video Inpainter (AAAI 2025 · similarity 0.89)
- Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation (ICLR 2025 · similarity 0.88)
- FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.