Research map / Video / Video segmentation and tracking
LaSe-E2V: Towards Language-guided Semantic-aware Event-to-Video Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LaSe-E2V, a language-guided text-conditional video diffusion model that reconstructs video from event streams, using event-guided spatio-temporal attention, an event-aware mask loss, and event-based noise initialization.
Problem
Event-to-video (E2V) reconstruction is ill-posed because events capture only local edge and motion information, so reconstructed videos suffer from artifacts, regional blur, and semantic ambiguity.
Approach
Adds language descriptions (from a tagging model) as semantic conditions to a text-to-video latent diffusion model, and makes it event-faithful via event-guided spatial/temporal attention, an event-aware mask loss on inter-frame differences, and event-accumulated noise initialization.
Key results
- On ECD, LaSe-E2V reaches MSE 0.023 vs 0.033 for HyperE2VID (about 30% reduction) and SSIM* 0.629 vs 0.576.
- On MVSEC, MSE 0.055 and SSIM* 0.342, best among compared methods; LPIPS 0.461 also best.
- Ablation on ECD: removing ESA raises MSE from 0.023 to 0.105 and drops SSIM from 0.629 to 0.468.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add text conditioning (ablation) | LaSe-E2V · ECD | SSIM: 0.567 -> 0.629 | Helped |
| add the previous-frame condition (ablation) | LaSe-E2V · ECD | SSIM: 0.474 -> 0.629 | Helped |
| add ESA instead of channel-wise event-feature concatenation (ablation) | LaSe-E2V · ECD | MSE: 0.105 -> 0.023 | Helped |
| add event-aware mask loss (ablation) | LaSe-E2V · ECD | MSE: 0.042 -> 0.023 | Helped |
| add event-based initialization (ablation) | LaSe-E2V · ECD | MSE: 0.043 -> 0.023 | Helped |
| use LaSe-E2V rather than HyperE2VID | ECD · ECD | MSE: 0.033 -> 0.023 | Helped |
| use LaSe-E2V rather than HyperE2VID | MVSEC · MVSEC | SSIM*: 0.315 -> 0.342 | Helped |
| use LaSe-E2V rather than HyperE2VID | HQF · HQF | MSE: 0.031 -> 0.034 | Hurt |
| use LaSe-E2V rather than E2VID+ | HQF · HQF | LPIPS: 0.252 -> 0.254 | Hurt |
Surprising findings
- Removing the previous-frame condition drops SSIM from 0.629 to 0.474 and raises MSE from 0.023 to 0.067, while the text-only ablation (no text) shows a 0.015 SSIM gain from text guidance.
Limitations
- Training data (synthetic plus limited real-world) is inadequate for data-intensive diffusion models, leading to artifacts on unseen scenes.
- Multiple denoising steps make inference slow.
Experimental setup
Datasets: MS-COCO BS-ERGB ECD MVSEC HQF HS-ERGB
Benchmarks: ECD MVSEC HQF HS-ERGB
Metrics: MSE SSIM SSIM* LPIPS
Baselines: E2VID FireNet E2VID+ FireNet+ SPADE-E2VID SSL-E2VID ET-Net HyperE2VID
Builds on: Stable Diffusion latent diffusion model E2VID HyperE2VID v2e
Closest papers by meaning
- E2HQV: High-Quality Video Generation from Event Camera via Theory-Inspired Model-Aided Deep Learning (AAAI 2024 · similarity 0.91)
- V2V: Scaling Event-Based Vision through Efficient Video-to-Voxel Simulation (NeurIPS 2025 · similarity 0.87)
- Talk2Event: Grounded Understanding of Dynamic Scenes from Event Cameras (NeurIPS 2025 · similarity 0.87)
- ESEG: Event-Based Segmentation Boosted by Explicit Edge-Semantic Guidance (AAAI 2025 · similarity 0.87)
- EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting (NeurIPS 2025 · similarity 0.87)
- EventGPT: Event Stream Understanding with Multimodal Large Language Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.