Research map / Image generation / Image restoration
RGB-Event ISP: The Dataset and Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the first event-RAW paired dataset (HVS-ISP, 3373 frames from a hybrid-vision sensor) with a controllable reference ISP, benchmarks learnable ISP methods with and without events, and proposes a simple event-guided UNet variant (EV-UNet).
Problem
Existing event-guided imaging work enhances RGB outputs post hoc and ignores the ISP stage, where RAW data from hybrid vision sensors is noisy and events could help; no dataset or benchmark pairs aligned RAW and events for event-guided ISP.
Approach
Collect pixel-aligned event and RAW data from a hybrid-vision sensor, generate reference RGB frames with a ColorChecker-calibrated conventional ISP, benchmark learnable ISPs, and fuse event voxel grids into a UNet encoder (EV-UNet) to test whether events help ISP.
Key results
- PyNet achieves the highest average outdoor PSNR of 32.47 dB, while EV-UNet improves over UNet outdoors from 28.17 to 30.11 dB PSNR.
- Event fusion hurts indoors: EV-UNet averages 26.04 dB PSNR versus UNet* at 31.76 dB on indoor scenes, attributed to flickering artificial lighting.
- The controllable ISP reaches average CIEDE2000 error of 5.84 and median 5.07 against the ColorChecker.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse events with UNet's encoder | UNet · HVS-ISP outdoor scenes (Average) | PSNR: 28.17 -> 30.11 | Helped |
| Fuse events with UNet's encoder | UNet · HVS-ISP indoor scenes | image reconstruction performance | Hurt |
| Use event-fusion methods under flickering indoor artificial lighting | EV-UNet · HVS-ISP indoor scenes | image reconstruction performance | Hurt |
Surprising findings
- Adding events to UNet improves outdoor performance but degrades indoor performance, attributed to flickering artificial light.
- AWNet, despite being a stage-wise method, performs poorly outdoors and overbrightens indoors when trained end-to-end, possibly due to sub-optimal training.
Limitations
- Dataset is small (24 videos, 3373 frames) because the HVS sensor is an unstable prototype.
- Controllable ISP requires a ColorChecker in the scene and cannot generalize to arbitrary scenes.
- Flickering artificial lighting degrades event-fusion performance indoors and is not addressed.
Experimental setup
Datasets: HVS-ISP Dataset
Benchmarks: HVS-ISP Dataset outdoor scenes HVS-ISP Dataset indoor scenes
Metrics: PSNR SSIM L1 CIEDE2000 CIELAB delta-ab parameters GFLOPS inference time
Baselines: PyNet PyNetCA InvertISP MV-ISPNet CameraNet AWNet Swin-Transformer UNet eSL-Net
Builds on: UNet PyNet AWNet CameraNet eSL-Net
Closest papers by meaning
- Towards Robust Event-guided Low-Light Image Enhancement: A Large-Scale Real-World Event-Image Dataset and Novel Approach (CVPR 2024 · similarity 0.85)
- EF-3DGS: Event-Aided Free-Trajectory 3D Gaussian Splatting (NeurIPS 2025 · similarity 0.84)
- ParamISP: Learned Forward and Inverse ISPs using Camera Parameters (CVPR 2024 · similarity 0.84)
- GS2E: Gaussian Splatting is an Effective Data Generator for Event Stream Generation (NeurIPS 2025 · similarity 0.84)
- LED: A Large-scale Real-world Paired Dataset for Event Camera Denoising (CVPR 2024 · similarity 0.84)
- Graph Neural Network Combining Event Stream and Periodic Aggregation for Low-Latency Event-based Vision (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.