Research map / Video / Video segmentation and tracking
VFIMamba: Video Frame Interpolation with State Space Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VFIMamba, the first video frame interpolation model built on selective state space models (S6/Mamba), using an interleaved-token Mixed-SSM Block and a curriculum learning schedule over motion magnitude.
Problem
Video frame interpolation needs inter-frame modeling with both a large receptive field for high-resolution, large-motion inputs and fast processing, which CNNs and attention models struggle to balance.
Approach
Replace convolution/attention inter-frame modeling with S6 blocks applied to an interleaved token sequence of two frames scanned in four directions, and train with a curriculum that gradually increases motion magnitude by resizing X-TRAIN data and doubling frame intervals while continuing Vimeo90K training.
Key results
- On X-TEST, VFIMamba reaches 33.34 dB at 2K and 32.15 dB at 4K, versus 32.38 dB and 31.35 dB for SGM-VFI (V+X).
- On low-resolution benchmarks, VFIMamba averages 34.30 dB PSNR versus 34.14 for EMA-VFI and 33.96 for SGM-VFI, with 0.94 TFLOPs at 720p.
- Curriculum learning improved RIFE and EMA-VFI-S on X-TEST and SNU-FILM while keeping Vimeo90K performance roughly unchanged.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use S6 instead of convolution for inter-frame modeling (ablation) | VFIMamba-S · X-TEST 2K | PSNR: 31.58 -> 32.84 | Helped |
| Use S6 instead of local attention for inter-frame modeling (ablation) | VFIMamba-S · X-TEST 2K | PSNR: 30.49 -> 32.84 | Helped |
| Remove S6 from the inter-frame modeling block (ablation) | VFIMamba-S · X-TEST 2K | PSNR: 32.84 -> 28.94 | Hurt |
| Use interleaved rather than sequential rearrangement in both scan directions (ablation) | VFIMamba-S · X-TEST 2K | PSNR: 28.07 -> 32.84 | Helped |
| Train with curriculum learning rather than without it (ablation) | VFIMamba-S · X-TEST 2K | PSNR: 30.82 -> 32.84 | Helped |
| Train with curriculum learning rather than without it (ablation) | VFIMamba-S · Vimeo90K | PSNR: 36.13 -> 36.12 | No clear effect |
| Train RIFE with curriculum learning (ablation) | RIFE · X-TEST 2K | PSNR: 31.10 -> 31.40 | Helped |
| Train EMA-VFI-S with curriculum learning (ablation) | EMA-VFI-S · X-TEST 2K | PSNR: 30.91 -> 31.15 | Helped |
Surprising findings
- Sequential rearrangement of two frames performs worse than interleaved rearrangement, with the interleaved variant giving up to about 4 dB gain on X-TEST 4K in ablation.
- Sequential learning (fine-tuning on X-TRAIN after Vimeo90K) degrades Vimeo90K performance, which motivates the curriculum design.
Limitations
- Still falls short of real-time requirements.
- Training restricted to Vimeo90K and X-TRAIN, which may cause overfitting and artifacts in real-world use.
- The frame generation module does not use SSMs; only inter-frame modeling is explored.
Experimental setup
Datasets: Vimeo90K X-TRAIN
Benchmarks: Vimeo90K UCF101 SNU-FILM X-TEST X-TEST-L Xiph
Metrics: PSNR SSIM FLOPs runtime parameters
Baselines: DAIN AdaCof CAIN Softsplat XVFI M2M-VFI RIFE IFRNet-L EMA-VFI AMT-G SGM-VFI BiFormer
Builds on: Mamba (S6) VMamba RIFE EMA-VFI FILM
Closest papers by meaning
- LC-Mamba: Local and Continuous Mamba with Shifted Windows for Frame Interpolation (CVPR 2025 · similarity 0.87)
- Disentangled Motion Modeling for Video Frame Interpolation (AAAI 2025 · similarity 0.87)
- BiM-VFI: Bidirectional Motion Field-Guided Frame Interpolation for Video with Non-uniform Motions (CVPR 2025 · similarity 0.87)
- IQ-VFI: Implicit Quadratic Motion Estimation for Video Frame Interpolation (CVPR 2024 · similarity 0.86)
- LDMVFI: Video Frame Interpolation with Latent Diffusion Models (AAAI 2024 · similarity 0.86)
- LAVIB: A Large-scale Video Interpolation Benchmark (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.