Research map / Video / Video segmentation and tracking
Generalizable Implicit Motion Modeling for Video Frame Interpolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GIMM, a generalizable implicit motion model that predicts arbitrary-timestep bilateral flows from bidirectional flows via a motion latent and SIREN-style coordinate network, plugged into a flow-based VFI pipeline (GIMM-VFI).
Problem
Flow-based video frame interpolation needs accurate motion modeling at arbitrary timestamps; existing approaches either linearly combine bidirectional flows or predict bilateral flows conditioned on discrete timesteps, which handles occlusion and complex spatiotemporal motion poorly and does not gen
Approach
Encode input-specific motion priors from bidirectional flows into a spatiotemporal motion latent (motion encoder, forward warping with splatting weights, latent refiner), then implicitly decode arbitrary-timestep normalized flows with a coordinate-based network conditioned on that latent, generalizing across videos rather than per-scene as in IFE.
Key results
- GIMM-VFI-R improves PSNR by 0.18 dB on XTest-2K and 0.67 dB on XTest-4K over the prior best EMA-VFI (32.71 vs 32.53 and 31.88 vs 31.21).
- On motion modeling, GIMM (RAFT-based) reaches 37.56 dB / 0.34 EPE on VTF and 30.45 dB / 2.68 EPE on VSF, beating Linear and End-to-End baselines.
- Integrating GIMM into TTVFI and IFRNet yields PSNR gains of about 1.0–1.6 dB on SNU-FILM-arb subsets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace INR with U-Net (ablation) | GIMM · VTF and VSF | PSNR: 37.56 -> 36.96; 30.45 -> 29.96 | Hurt |
| replace INR with meta-learning weight modulation (ablation) | GIMM · VTF and VSF | PSNR: 37.56 -> 30.19; 30.45 -> 24.50 | Hurt |
| add forward warping of motion features (ablation) | GIMM · VTF and VSF | EPE: 0.37 -> 0.34; 2.88 -> 2.68 | Helped |
| add coordinate-based implicit modeling (ablation) | GIMM · VTF and VSF | EPE: 0.39 -> 0.34; 2.81 -> 2.68 | Helped |
| add the motion encoder (ablation) | GIMM · VTF and VSF | EPE: 0.42 -> 0.34; 2.85 -> 2.68 | Helped |
| add the latent refiner (ablation) | GIMM · VTF and VSF | PSNR: 37.03 -> 37.56; 30.02 -> 30.45 | Helped |
| use 3D spatiotemporal coordinates instead of temporal coordinates only (ablation) | GIMM · VTF and VSF | PSNR: 37.40 -> 37.56; 30.39 -> 30.45 | Helped |
Surprising findings
- Replacing the INR with a timestep-conditioned U-Net of comparable channels degrades motion modeling, and a transformer meta-learner variant performs worse despite having over 170X more parameters.
Limitations
- Inherits limitations and computational cost of the pre-trained flow estimator
- Accepts only two consecutive frames as input, which is unfavorable for large nonlinear motion
Experimental setup
Datasets: Vimeo90K SNU-FILM X4K-1000FPS (XTest)
Benchmarks: Vimeo-Triplet-Flow Vimeo-Septuplet-Flow SNU-FILM-arb XTest-2K XTest-4K
Metrics: PSNR EPE LPIPS FID
Baselines: RIFE M2M IFRNet AMT UPR-Net EMA-VFI CURE BMBC
Builds on: IFE AMT RAFT FlowFormer SIREN
Closest papers by meaning
- BiM-VFI: Bidirectional Motion Field-Guided Frame Interpolation for Video with Non-uniform Motions (CVPR 2025 · similarity 0.89)
- Disentangled Motion Modeling for Video Frame Interpolation (AAAI 2025 · similarity 0.88)
- IQ-VFI: Implicit Quadratic Motion Estimation for Video Frame Interpolation (CVPR 2024 · similarity 0.87)
- Implicit Motion Function (CVPR 2024 · similarity 0.87)
- Sparse Global Matching for Video Frame Interpolation with Large Motion (CVPR 2024 · similarity 0.86)
- StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.