Research map / Video / Video segmentation and tracking
E2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes E2HQV, a model-aided deep learning framework for events-to-video reconstruction that estimates key parameters of a theory-inspired E2V model plus a temporal shift embedding module to handle recurrent state-resets.
Problem
Events-to-video (E2V) reconstruction from event streams is purely data-driven in existing methods, leading to poor detailed texture recovery in complex scenes and sensitivity to unknown, time-varying event thresholds and noise.
Approach
Derive a physics-based relation between consecutive frames and inter-frame event counts (log-intensity change with polarity thresholds), then train a network to estimate the thresholds and event counts and generate frames via this model; add a temporal shift embedding that gives pixelwise, reset-interval-aware embeddings to handle recurrent state resets.
Key results
- E2HQV reduces MSE by about 40.4%, 70.1%, and 40.6% relative to the second-best method (SSL-E2VID) across the three datasets (reported in text).
- E2HQV achieves SSIM of 0.682 (IJRR), 0.421 (MVSEC), and 0.671 (HQF), the best among compared methods.
- E2HQV has 7.82M parameters and 18.35 GFLOPs, versus 22.18M and 145.12 GFLOPs for ET-Net.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use E2HQV rather than SSL-E2VID | E2V reconstruction · IJRR | MSE: 0.046 -> 0.028 | Helped |
| Use E2HQV rather than ET-Net | E2V reconstruction · IJRR | SSIM: 0.617 -> 0.682 | Helped |
| Use E2HQV rather than ET-Net | E2V reconstruction · IJRR | LPIPS: 0.224 -> 0.196 | Helped |
| Use E2HQV rather than SSL-E2VID | E2V reconstruction · MVSEC | MSE: 0.062 -> 0.032 | Helped |
| Use E2HQV rather than ET-Net | E2V reconstruction · MVSEC | SSIM: 0.380 -> 0.421 | Helped |
| Use E2HQV rather than ET-Net | E2V reconstruction · MVSEC | LPIPS: 0.489 -> 0.460 | Helped |
| Use E2HQV rather than ET-Net | E2V reconstruction · HQF | MSE: 0.032 -> 0.019 | Helped |
| Use E2HQV rather than ET-Net | E2V reconstruction · HQF | SSIM: 0.658 -> 0.671 | Helped |
| Use E2HQV rather than E2VID+ | E2V reconstruction · HQF | LPIPS: 0.252 -> 0.261 | Hurt |
| Add REFE to MAFG (ablation) | E2HQV · IJRR | MSE: 0.080 -> 0.051 | Helped |
| Add TSEM to MAFG (ablation) | E2HQV · IJRR | MSE: 0.080 -> 0.032 | Helped |
| Add REFE to MAFG (ablation) | E2HQV · MVSEC | MSE: 0.062 -> 0.064 | Hurt |
Surprising findings
- On HQF, the LPIPS of E2HQV (0.261) is slightly worse than E2VID+ (0.252), so it does not win on all metrics.
- The SSL-E2VID baseline has the best MSE on MVSEC after E2HQV-level comparisons, but its SSIM on HQF is very low (0.295).
Limitations
- Baselines trained only on simulated data and tested without fine-tuning on real datasets.
- Thresholds and event counts are estimated by a network, and the theory-inspired model is an approximation (assumes constant thresholds and linear intensity-frame relation).
Experimental setup
Datasets: MS-COCO (ESIM-simulated events) IJRR MVSEC HQF
Benchmarks: IJRR MVSEC HQF
Metrics: MSE SSIM LPIPS
Baselines: E2VID FireNet E2VID+ FireNet+ SPADE-E2VID SSL-E2VID ET-Net
Builds on: E2VID EfficientNetV2 ESIM VoxelGrid
Closest papers by meaning
- LaSe-E2V: Towards Language-guided Semantic-aware Event-to-Video Reconstruction (NeurIPS 2024 · similarity 0.91)
- V2V: Scaling Event-Based Vision through Efficient Video-to-Voxel Simulation (NeurIPS 2025 · similarity 0.91)
- EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting (NeurIPS 2025 · similarity 0.86)
- EventZoom: A Progressive Approach to Event-Based Data Augmentation for Enhanced Neuromorphic Vision (AAAI 2025 · similarity 0.86)
- EventGPT: Event Stream Understanding with Multimodal Large Language Models (CVPR 2025 · similarity 0.86)
- LED: A Large-scale Real-world Paired Dataset for Event Camera Denoising (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.