Research map / Video / Video segmentation and tracking
V2V: Scaling Event-Based Vision through Efficient Video-to-Voxel Simulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Video-to-Voxel (V2V), which converts video frames directly into discrete event voxel grids on the fly, avoiding event stream storage, and uses it to scale event-based video reconstruction and optical flow training to 10K WebVid videos.
Problem
Event-based vision models lack large-scale training data because synthetic event stream generation is storage- and I/O-intensive and real event data is scarce; video-based and model-based simulators also suffer from temporal-resolution gaps and limited scene realism.
Approach
Skip explicit event stream synthesis: simulate discrete event voxels directly from consecutive video frames with randomly sampled event camera parameters (thresholds, noise, hot pixels) re-drawn each training iteration, exploiting that discrete voxels discard intra-bin timing that cannot be reliably simulated from low-FPS video.
Key results
- V2V-trained E2VID on WebVid10K improves HQF SSIM from 0.638 (ESIM-pretrained) to 0.676 and EVAID SSIM from 0.526 to 0.663.
- Storage for the WebVid datasets is reduced by about 150x (279 GB to 20 GB for the dataset scale reported), with WebVid10K using 6.9% of ESIM-Voxel disk space while providing 35.7x more scene diversity.
- V2V-EvFlow exceeds EvFlow+ on all MVSEC dense and sparse AEE and 3PE metrics, e.g. indoor_flying1 dense AEE 0.732 vs 0.967.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train E2VID with V2V on WebVid10K instead of ESIM pretraining | E2VID · HQF | SSIM: 0.638 -> 0.676 | Helped |
| Train E2VID with V2V on WebVid10K instead of ESIM pretraining | E2VID · EVAID | SSIM: 0.526 -> 0.663 | Helped |
| Train HyperE2VID with V2V on WebVid10K instead of ESIM pretraining | HyperE2VID · EVAID | SSIM: 0.533 -> 0.654 | Helped |
| Train ETNet with V2V on WebVid10K* instead of ESIM pretraining | ETNet · HQF | SSIM: 0.642 -> 0.662 | Helped |
| Train EvFlow with V2V on WebVid10K rather than EvFlow+ training | EvFlow · MVSEC indoor_flying1 | AEE (D): 0.967 -> 0.732 | Helped |
| Train EvFlow with V2V on WebVid10K rather than EvFlow+ training | EvFlow · MVSEC outdoor_day1 | AEE (D): 2.903 -> 2.114 | Helped |
| Use discrete instead of interpolated voxels (ablation) | E2VID · HQF | SSIM: 0.599 -> 0.608 | Helped |
| Randomize threshold configurations per iteration rather than fixing thresholds per video (ablation) | E2VID on WebVid10K · EVAID | SSIM: 0.644 -> 0.663 | Helped |
| Filter WebVid videos for artifacts, blank content, and blur (ablation) | E2VID on WebVid10K · HQF | SSIM: 0.676 -> 0.662 | Hurt |
| Degrade the dynamic range of training videos (ablation) | E2VID on WebVid10K · HQF | SSIM: 0.676 -> 0.647 | Hurt |
| Use Alex LPIPS plus full temporal consistency instead of VGG+L1+half consistency (ablation) | E2VID on WebVid10K · EVAID | SSIM: 0.663 -> 0.604 | Hurt |
| Replace ESIM ground-truth flow with RAFT-Small predicted flow in temporal consistency loss (ablation) | E2VID on ESIM-280-Intp. · HQF | SSIM: 0.599 -> 0.613 | No clear effect |
Surprising findings
- Filtering WebVid with a VLM to remove blur, CG, and artifacts produced slightly worse results than the unfiltered set, attributed to reduced diversity.
- Using the Alex LPIPS loss caused patterned artifacts on WebVid, so a VGG-based LPIPS with L1 was used instead.
Limitations
- Video decoding is CPU-intensive and may become a bottleneck on systems with strong GPUs and weak CPUs.
- Only applicable to representations that discard intra-bin temporal information, such as discrete voxels; not to spiking neural networks or interpolated voxels.
- Real test sets (DAVIS-based HQF, IJRR, MVSEC) have noisy frames and hot/dead pixels, making metrics unreliable; checkpoint selection strongly affects results.
Experimental setup
Datasets: WebVid ESIM-280 HQF EVAID IJRR MVSEC
Benchmarks: HQF EVAID IJRR MVSEC
Metrics: MSE SSIM LPIPS AEE 3PE
Baselines: E2VID E2VID+ HyperE2VID ETNet EvFlow+ Zeros
Builds on: E2VID E2VID+ EvFlow V2E RAFT
Closest papers by meaning
- E2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learning (AAAI 2024 · similarity 0.91)
- LaSe-E2V: Towards Language-guided Semantic-aware Event-to-Video Reconstruction (NeurIPS 2024 · similarity 0.87)
- EventZoom: A Progressive Approach to Event-Based Data Augmentation for Enhanced Neuromorphic Vision (AAAI 2025 · similarity 0.87)
- EventFly: Event Camera Perception from Ground to the Sky (CVPR 2025 · similarity 0.86)
- Efficient Meshflow and Optical Flow Estimation from Event Cameras (CVPR 2024 · similarity 0.85)
- RE-VLM: Event-Augmented Vision-Language Model for Scene Understanding (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.