Research map / Video / Video segmentation and tracking
Video Frame Interpolation via Direct Synthesis with the Event-based Reference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a pure synthesis-based event-based VFI framework that reconstructs an event-based structural reference for the missing frame and aligns keyframes to it via a deformable E-PCD module, followed by a transformer refinement decoder.
Problem
Event-based video frame interpolation methods rely on motion estimation and warping, which fail under severe occlusion, missing moving objects, and large nonlinear motion.
Approach
Instead of using events only to refine motion, reconstruct an event-based reference of the target frame (with an event-aware erosion/dilation mask suppressing unreliable regions) and synthesize the interpolated frame by bidirectionally aligning keyframes to this reference with an event-guided deformable module (E-PCD).
Key results
- On Vimeo90k-Triplet the method reaches 39.17 dB PSNR / 0.977 SSIM, above retrained CBMNet-L (37.69 dB) under identical training.
- On SNU-FILM Extreme, the method reaches 31.46 dB / 0.893 versus 28.99 dB for CBMNet-L under the same training strategy.
- On HQF with 3 skipped frames, the method is about 1 dB PSNR above CBMNet-L (35.89 vs 34.77 at 1 frame; 34.27 vs 33.08 at 3 frames).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed method instead of the strongest reported baseline | Video frame interpolation · Vimeo90k-Triplet | PSNR (dB): 37.69 -> 39.17 | Helped |
| Use the proposed method instead of the strongest reported baseline | Video frame interpolation · GoPro 1 frame and 7 frame | PSNR (dB): 38.15 -> 36.95; 37.05 -> 35.77 | Hurt |
| Use the proposed method instead of the strongest reported baseline | Video frame interpolation · GoPro 15 frame | PSNR (dB): 35.46 -> 35.77 | Helped |
| Use the proposed method instead of the strongest reported baseline | Video frame interpolation · SNU-FILM Hard and Extreme | PSNR (dB): 31.75 -> 33.04; 28.99 -> 31.46 | Helped |
| Use the proposed method instead of the strongest reported baseline | Video frame interpolation · Real-world datasets: HQF and HS-ERGB | PSNR (dB): 34.77 -> 35.89; 33.08 -> 34.27; 34.17 -> 34.60; 33.96 -> 34.33; 33.64 -> 34.19; 33.15 -> 33.56 | Helped |
| Add event guidance and multi-level reference fusion to the PCD alignment module (ablation) | E-PCD on HS-ERGB far · HS-ERGB far | PSNR: 29.92 -> 33.56 | Helped |
| Add the event-aware reconstruction strategy (ablation) | Event-based reference reconstruction on HS-ERGB far · HS-ERGB far | PSNR: 32.47 -> 33.56 | Helped |
Surprising findings
- Without explicit motion estimation, the pure synthesis approach beats motion-based E-VFI methods on extreme-motion SNU-FILM splits.
- The method is reported to produce interpolated content not visible in the keyframes, such as plate edges, via the event-based reference.
Limitations
- On GoPro, the method trails CBMNet-L trained on the full GoPro training set (with the same training strategy it leads).
- Event-aware masking relies on hand-set kernel sizes and threshold (3, 5, 6), and reconstruction of static or low-texture regions remains unreliable.
- Real-world evaluation requires fine-tuning on the target datasets.
Experimental setup
Datasets: Vimeo90k-Septuplet GoPro ESIM simulated events
Benchmarks: Vimeo90k-Triplet GoPro SNU-FILM HQF HS-ERGB
Metrics: PSNR SSIM
Baselines: BMBC RIFE UPR-Net VFIT-B A2OF TimeLens CBMNet-L TimeReplayer SuperFast
Builds on: PCD module (EDVR) TimeLens CBMNet
Closest papers by meaning
- Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation (CVPR 2025 · similarity 0.91)
- TTA-EVF: Test-Time Adaptation for Event-based Video Frame Interpolation via Reliable Pixel and Sample Estimation (CVPR 2024 · similarity 0.90)
- TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion (CVPR 2025 · similarity 0.89)
- LaSe-E2V: Towards Language-guided Semantic-aware Event-to-Video Reconstruction (NeurIPS 2024 · similarity 0.86)
- Video Frame Prediction from a Single Image and Events (AAAI 2024 · similarity 0.86)
- Perception-Oriented Video Frame Interpolation via Asymmetric Blending (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.