Research map / Video / Video segmentation and tracking
Disentangled Motion Modeling for Video Frame Interpolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MoMo, a diffusion-based video frame interpolation method that generates bi-directional optical flows with a motion diffusion model using a novel convex-upsampling U-Net, trained in a two-stage disentangled scheme.
Problem
Video frame interpolation methods trained with reconstruction losses yield blurry, perceptually poor results, while existing generative VFI methods model pixels or latents at high computational cost.
Approach
Disentangle training: first train a frame synthesis network and fine-tune a RAFT flow model, then train a diffusion model to generate bi-directional intermediate flows (x0-prediction, L1 loss, 1/8-resolution U-Net with convex upsampling) that the synthesis network uses to render frames.
Key results
- On SNU-FILM-extreme, MoMo reaches LPIPS 0.0872 and DISTS 0.0433, best among compared methods (FILM-Ls 0.0889/0.0448).
- On Xiph-4K, MoMo achieves LPIPS 0.0631 and DISTS 0.0274, versus 0.0703/0.0385 for the next best perceptual method FILM-Ls.
- Convex-upsampling U-Net reduces FLOPs ~7.2x (8.08 to 1.12 TFLOPs) and runtime to 145.49 ms per frame, about 70x faster than LDMVFI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoMo rather than the strongest competing perceptual VFI methods | VFI · SNU-FILM easy, medium, hard, and extreme | LPIPS: 0.0120 -> 0.0111; 0.0213 -> 0.0202; 0.0429 -> 0.0419; 0.0889 -> 0.0872 | Helped |
| Use MoMo rather than the strongest competing perceptual VFI methods | VFI · Xiph-2K and Xiph-4K | DISTS: 0.0153 -> 0.0119; 0.0331 -> 0.0274 | Helped |
| Use MoMo rather than FILM-Ls | VFI · Middlebury and Vimeo90k | LPIPS: 0.0093 -> 0.0094; 0.0131 -> 0.0136 | Mixed |
| Use alternating training with the synthesis network fixed when fine-tuning the flow teacher (ablation) | MoMo · SNU-FILM-hard | LPIPS: 0.0445 -> 0.0419; 0.0475 -> 0.0419 | Helped |
| Use x0-prediction rather than epsilon-prediction (ablation) | MoMo motion diffusion · SNU-FILM-hard | LPIPS: 0.4090 -> 0.0460 | Helped |
| Use Convex-Up U-Net rather than standard U-Net (ablation) | MoMo motion diffusion · SNU-FILM-hard | LPIPS: 0.0460 -> 0.0425 | Helped |
| Avoid SNR-weighting the x0-prediction loss (ablation) | MoMo motion diffusion · SNU-FILM-hard | LPIPS: 0.0463 -> 0.0425 | Helped |
| Train the motion diffusion model longer (ablation) | MoMo motion diffusion · SNU-FILM-hard | LPIPS: 0.0425 -> 0.0419 | Helped |
| Increase denoising steps from 1 to 8 (ablation) | MoMo motion diffusion · SNU-FILM-extreme | LPIPS: 0.0892 -> 0.0872 | Helped |
| Increase denoising steps beyond 8 (ablation) | MoMo motion diffusion · SNU-FILM-extreme | LPIPS: 0.0872 -> 0.0872; 0.0872 -> 0.0874 | No clear effect |
Surprising findings
- Contrary to common practice, x0-prediction with L1 loss outperformed epsilon-prediction for flow diffusion; epsilon-prediction with flows performed very poorly (LPIPS 0.4090).
Limitations
- Motion diffusion model is trained at 256 resolution and may degrade on high-resolution videos with large motion, requiring resize-and-upsample inference.
- Reports lower PSNR/SSIM than several reconstruction-loss methods, so gains are mainly perceptual.
- On SNU-FILM-hard and extreme subsets, MoMo does not lead on PSNR/SSIM; the method was evaluated mainly with perceptual metrics.
Experimental setup
Datasets: Vimeo90k
Benchmarks: SNU-FILM Middlebury (others-set) Vimeo90k Xiph-2K Xiph-4K
Metrics: LPIPS DISTS PSNR SSIM
Baselines: ABME XVFI IFRNet RIFE FILM AMT-G EMA-VFI UPRNet CAIN LDMVFI PerVFI
Builds on: RAFT LDMVFI FILM convex upsampling DDPM
Closest papers by meaning
- Perception-Oriented Video Frame Interpolation via Asymmetric Blending (CVPR 2024 · similarity 0.90)
- Hierarchical Flow Diffusion for Efficient Frame Interpolation (CVPR 2025 · similarity 0.89)
- EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation (CVPR 2025 · similarity 0.89)
- BiM-VFI: Bidirectional Motion Field-Guided Frame Interpolation for Video with Non-uniform Motions (CVPR 2025 · similarity 0.88)
- LDMVFI: Video Frame Interpolation with Latent Diffusion Models (AAAI 2024 · similarity 0.88)
- Generalizable Implicit Motion Modeling for Video Frame Interpolation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.