Research map / Video / Video segmentation and tracking
EVS-assisted Joint Deblurring Rolling-Shutter Correction and Video Frame Interpolation through Sensor Inverse Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Jointly deblurs, rolling-shutter corrects, and interpolates hybrid CIS-EVS frames up to 10,000 FPS by solving a per-pixel nonlinear inverse sensor model that includes EVS pixel (VFE) latency, readout latency and refractory period, followed by a NAFNet-based refinement network.
Problem
Enhancing blurry, rolling-shutter-distorted CIS video (120-150 FPS) into high-frame-rate sharp video using EVS data, while EVS nonidealities (pixel/front-end latency, readout latency, refractory period) degrade fusion quality, especially under low light and fast motion.
Approach
Explicitly models EVS pixel latency (VFE low-pass delay), readout latency and refractory period in the event measurement equations, then solves the resulting nonlinear system per pixel via graph optimization jointly over consecutive CIS frames row-by-row, with a spatially-attentive NAFNet refinement trained with L1, VGG perceptual and Gram losses.
Key results
- Average PSNR on the simulation dataset is 27.51 dB for the proposed method versus 23.42 dB for the next best (REFID's column is 22.85 dB; Time Lens 21.68 dB), with the refinement-free variant at 24.09 dB.
- The paper reports up to 4 dB PSNR improvement and 12% LPIPS improvement over state-of-the-art methods.
- On the rotating disk at 1000 lx and 292 rpm, the proposed method achieves the lowest BEW (0.7) among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the proposed sensor inverse modeling and refinement method rather than the strongest average-PSNR baseline, Time Lens | Video reconstruction · Simulation dataset, Average | PSNR: 23.42 -> 27.51 | Helped |
| Add the proposed method rather than FILM | Video reconstruction · Simulation dataset, Average | LPIPS: 0.182 -> 0.16 | Helped |
| Add refinement network (ablation) | Ours · Basketball | PSNR: 24.01 -> 27.23 | Helped |
| Add refinement network (ablation) | Ours · Slingshot egg | PSNR: 25.23 -> 32.85 | Helped |
| Add refinement network (ablation) | Ours · Basketball | LPIPS: 0.217 -> 0.155 | Helped |
| Add refinement network (ablation) | Ours · Slingshot egg | LPIPS: 0.100 -> 0.106 | Hurt |
| Enable VFE delay compensation (ablation) | Ours · Basketball | PSNR: 22.95 -> 23.47 | Helped |
| Enable VFE delay compensation (ablation) | Ours · Slingshot egg | PSNR: 23.57 -> 24.00 | Helped |
| Add readout latency and refractory period compensation to VFE delay compensation (ablation) | Ours · Basketball | PSNR: 23.47 -> 23.56 | Helped |
| Add readout latency and refractory period compensation to VFE delay compensation (ablation) | Ours · Slingshot egg | PSNR: 24.00 -> 23.71 | Hurt |
| Add refinement network to VFE+RL+RP compensation (ablation) | Ours · Basketball | PSNR: 23.56 -> 23.64 | Helped |
| Add refinement network to VFE+RL+RP compensation (ablation) | Ours · Slingshot egg | PSNR: 23.71 -> 29.04 | Helped |
Surprising findings
- Adding readout latency and refractory period compensation improves basketball but degrades slingshot egg relative to VFE-only compensation, attributed to violated locally-constant slope assumption.
- The method does not improve image quality in bright or slow-motion scenes such as running man and spinning umbrella.
Limitations
- Image quality gains are subtle for slow objects or bright scenes
- Residual dark tail/overshoot artifacts along fast-moving pellet trajectories
- Measured HS-ERGB results use a different EVS and dual-camera setup, reducing gains
Experimental setup
Datasets: simulated CIS-EVS hybrid sensor data (simulator from [26]) HS-ERGB measured hybrid CIS-EVS natural scene dataset measured rotating disk dataset
Benchmarks: proposed simulation dataset HS-ERGB measured natural scene dataset rotating disk dataset
Metrics: PSNR LPIPS BEW
Baselines: EDI AKF Time Lens FILM CBMNet EvUnroll REFID
Builds on: EDI NAFNet FILM Time Lens EvUnroll
Closest papers by meaning
- EvShutter: Transforming Events for Unconstrained Rolling Shutter Correction (CVPR 2023 · similarity 0.90)
- EvUnroll: Neuromorphic Events Based Rolling Shutter Image Correction (CVPR 2022 · similarity 0.88)
- Latency Correction for Event-guided Deblurring and Frame Interpolation (CVPR 2024 · similarity 0.85)
- Single Image Rolling Shutter Removal with Diffusion Models (AAAI 2025 · similarity 0.84)
- EvFocus: Learning to Reconstruct Sharp Images from Out-of-Focus Event Streams (ICML 2025 · similarity 0.82)
- Frequency-aware Event-based Video Deblurring for Real-World Motion Blur (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.