Research map / Video / Video segmentation and tracking
Video Frame Prediction from a Single Image and Events
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Predicts future video frames from a single RGB image plus following events using a lightweight pyramidal coarse-to-fine model with cross-modal attention and hole inpainting, running at 720P in 25 ms.
Problem
Video frame prediction from past frames is limited by fixed frame rate inputs, struggling with complex dynamics and flexible prediction time intervals.
Approach
Replace multi-frame input with a single image and subsequent events; use symmetrical cross-modal attention between image and event features, a joint flow-and-frame coarse-to-fine decoder, and an inpainting module that fills forward-warping holes with synthesized features, followed by weighted fusion.
Key results
- On GoPro with 15 frames, the model reaches 27.71 dB PSNR, improving over OVP by up to 3 dB and over IFRNet by up to 2.93 dB.
- Predicts a 720P frame in 0.024 s on an RTX2080Ti with an 8.3 MB model, the only method meeting real-time demands in the comparison.
- On DSEC 3-frame VFP, PSNR is 26.61 dB versus 25.59 dB for DCEIFlow* and 20.59 dB for EDI.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| predict from one image and following events with the proposed model rather than using Time Lens | Video frame prediction · GoPro, 7 frames | PSNR (dB): 32.66 -> 29.73 | Hurt |
| predict from one image and following events with the proposed model rather than using Time Lens | Video frame prediction · GoPro, 15 frames | PSNR (dB): 29.81 -> 27.71 | Hurt |
| predict from one image and following events with the proposed model rather than using Time Lens | Video frame prediction · HS-ERGB, 7 frames | PSNR (dB): 32.12 -> 28.07 | Hurt |
| add cross-modal attention augmentation (ablation) | Proposed model · GoPro test set | PSNR: 28.58 -> 30.80 | Helped |
| apply attention augmentation on the 1st and 4th layers rather than only one pyramid layer (ablation) | Proposed model · GoPro test set | PSNR: 29.32, 29.08, 29.15, 29.21 -> 30.80 | Helped |
| use forward flow with inpainting instead of forward flow without inpainting (ablation) | Proposed model · GoPro test set | PSNR: 26.14 -> 30.80 | Helped |
| estimate backward rather than forward flow with inpainting (ablation) | Proposed model · GoPro test set | PSNR: 30.80 -> 30.84 | No clear effect |
| estimate backward rather than forward flow with inpainting (ablation) | Proposed model · GoPro test set | SSIM: 0.95 -> 0.94 | Hurt |
| add task-oriented flow loss (ablation) | Proposed model · GoPro test set | PSNR: 29.75 -> 30.80 | Helped |
| add feature consistency loss (ablation) | Proposed model · GoPro test set | PSNR: 28.45 -> 30.80 | Helped |
| add Charbonnier reconstruction loss (ablation) | Proposed model · GoPro test set | PSNR: 28.69 -> 30.80 | Helped |
| add LPIPS reconstruction loss (ablation) | Proposed model · GoPro test set | PSNR: 30.52 -> 30.80 | Helped |
Surprising findings
- Estimating residual intensity instead of absolute intensity for the synthesized frame gave no meaningful gain (30.83 vs 30.80 dB PSNR).
Limitations
- Performance is inferior to Time Lens, which uses a second frame after the target time.
- Forward warping produces holes that require inpainting; backward flow gives higher PSNR but ghosting.
- Joint flow and frame learning needs a two-stage training scheme because of a chicken-and-egg dependency.
Experimental setup
Datasets: Vimeo90k septuplet GoPro HS-ERGB DSEC
Benchmarks: GoPro HS-ERGB DSEC
Metrics: PSNR SSIM runtime model size
Baselines: IFRNet Time Lens E2VID HyperE2VID DCEIFlow OVP DMVFN EDI EVDI
Builds on: EDI Time Lens DCEIFlow Vid2E Census loss
Closest papers by meaning
- Video Frame Interpolation via Direct Synthesis with the Event-based Reference (CVPR 2024 · similarity 0.86)
- TTA-EVF: Test-Time Adaptation for Event-based Video Frame Interpolation via Reliable Pixel and Sample Estimation (CVPR 2024 · similarity 0.85)
- TimeTracker: Event-based Continuous Point Tracking for Video Frame Interpolation with Non-linear Motion (CVPR 2025 · similarity 0.85)
- StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences (NeurIPS 2024 · similarity 0.85)
- MESSENGER: Memory-Enhanced Sequential Scene Flow Estimation via Autoregressive Next-Frame Forecasting (NeurIPS 2026 · similarity 0.84)
- Repurposing Pre-trained Video Diffusion Models for Event-based Video Interpolation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.