Research map / Video / Video segmentation and tracking
Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a frozen pre-trained Stable Video Diffusion model with a trainable ControlNet-style event branch and test-time tiled denoising plus two-side fusion for event-based video frame interpolation.
Problem
Event-based video frame interpolation (EVFI) methods are trained on small paired event-frame datasets and generalize poorly to unseen data and motion distributions; the goal is to leverage internet-scale pre-trained video diffusion priors for EVFI.
Approach
Keep a pre-trained video diffusion denoiser frozen, add a trainable copy of its blocks conditioned on multi-stack event representations (ControlNet-style) to avoid catastrophic forgetting, then at inference use upsampled per-tile denoising and a two-side fusion of forward (start-frame) and backward (end-frame) denoised latents for interpolation.
Key results
- On BS-ERGB (3 skips), RE-VDM reaches 27.74 PSNR / 0.88 SSIM versus 26.24 PSNR for CBMNet-Large and 26.01 for EMA-VFI.
- On the self-collected Clear-Motion test set (15 skips), RE-VDM achieves 22.94 PSNR, above CBMNet-Large (22.26) and GIMM-VFI-R-P (17.77).
- On HQF, RE-VDM achieves 29.04 PSNR / 0.89 SSIM, the best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Adapt Stable Video Diffusion with event-based control | RE-VDM · BS-ERGB (3 skips) | PSNR: 26.24 -> 27.74 | Helped |
| Adapt Stable Video Diffusion with event-based control | RE-VDM · BS-ERGB (3 skips) | SSIM: 0.86 -> 0.88 | Helped |
| Adapt Stable Video Diffusion with event-based control | RE-VDM · BS-ERGB (3 skips) | LPIPS: 0.08 -> 0.12 | Hurt |
| Adapt Stable Video Diffusion with event-based control | RE-VDM · HQF (3 skips) | PSNR: 28.73 -> 29.04 | Helped |
| Adapt Stable Video Diffusion with event-based control | RE-VDM · HQF (3 skips) | SSIM: 0.84 -> 0.89 | Helped |
| Adapt Stable Video Diffusion with event-based control | RE-VDM · HQF (3 skips) | LPIPS: 0.04 -> 0.06 | Hurt |
| Adapt Stable Video Diffusion with event-based control | RE-VDM · Clear-Motion (15 skips) | PSNR: 22.26 -> 22.94 | Helped |
| Adapt Stable Video Diffusion with event-based control | RE-VDM · Clear-Motion (15 skips) | SSIM: 0.86 -> 0.88 | Helped |
| Adapt Stable Video Diffusion with event-based control | RE-VDM · Clear-Motion (15 skips) | LPIPS: 0.26 -> 0.37 | Hurt |
| Add Two-side Fusion of forward and backward denoised latents (ablation) | RE-VDM · EVFI | frame interpolation consistency | Helped |
| Add upsampling in Per-tile Denoising and Fusion (ablation) | RE-VDM · EVFI | appearance fidelity | Helped |
Surprising findings
- Frame-only methods can achieve better LPIPS than the proposed method on some datasets (e.g., GIMM-VFI-R-P 0.08 vs 0.12 on BS-ERGB), despite lower PSNR/SSIM.
- Pre-trained video diffusion model alone (Time Reversal) performs poorly on real-world interpolation, showing bias toward animation-style generation.
Limitations
- VAE encode/decode causes color shifts and loss of small details.
- Adapted model smooths certain textures; high upsampling factors needed, which is costly.
- Diffusion-based inference is more memory-intensive and time-consuming than other methods.
Experimental setup
Datasets: BS-ERGB HQF Clear-Motion
Benchmarks: BS-ERGB HQF Clear-Motion
Metrics: PSNR SSIM LPIPS
Baselines: Time Reversal DynamiCrafter RIFE PerVFI InterpAny-Clearer EMA-VFI GIMM-VFI-R-P CBMNet-Large
Builds on: ControlNet Stable Video Diffusion MultiDiffusion Time Reversal multi-stack event representation
Closest papers by meaning
- TTA-EVF: Test-Time Adaptation for Event-based Video Frame Interpolation via Reliable Pixel and Sample Estimation (CVPR 2024 · similarity 0.91)
- Video Frame Interpolation via Direct Synthesis with the Event-based Reference (CVPR 2024 · similarity 0.91)
- EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation (CVPR 2025 · similarity 0.89)
- LDMVFI: Video Frame Interpolation with Latent Diffusion Models (AAAI 2024 · similarity 0.89)
- TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion (CVPR 2025 · similarity 0.88)
- Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.