Research map / Video / Video segmentation and tracking
Towards HDR and HFR Video from Rolling-Mixed-Bit Spikings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes rolling-mixed-bit (RMB) spiking readout that interleaves multi-bit rows within single-bit spikings, and RMB-Net with cross-bit and cross-time attention to reconstruct HDR and HFR video from spiking cameras.
Problem
Spiking cameras give HDR and high temporal resolution but single-bit spikings limit dynamic range; increasing bit depth raises readout data volume and hurts high frame rate reconstruction.
Approach
A rolling readout interleaves a few rows of 3-bit spikings into single-bit spiking planes (about 1.6% extra data), and a two-stage attention network (cross-bit attention then cross-time attention, with ConvLSTM) fuses the mixed-bit streams into HDR/HFR frames.
Key results
- On synthetic data RMB-Net reaches 24.06 PSNR and 0.895 SSIM, versus 19.22 for the best prior baseline (GC20 at 19.22 PSNR) and 26.62 PSNR for Pure-M (pure 3-bit).
- RMB readout with 4 rows of 3-bit spikings per 500-row plane increases data volume by only 1.6% over single-bit spikings, versus 3x for pure 3-bit.
- Ablations show removing cross-bit attention drops PSNR to 22.29 and removing cross-time attention drops it to 21.81 (full model 24.06).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RMB-Net instead of GC20 | HDR video reconstruction from RMB spikings · Synthetic data | PSNR: 19.22 -> 24.06 | Helped |
| use RMB-Net instead of GC20 | HDR video reconstruction from RMB spikings · Synthetic data | SSIM: 0.800 -> 0.895 | Helped |
| use RMB-Net instead of Spk2ImgNet | HDR video reconstruction from RMB spikings · Synthetic data | HDR-VDP3: 7.664 -> 8.103 | Helped |
| use RMB-Net instead of GC20 | HDR video reconstruction from RMB spikings · Synthetic data | HDR-VQM: 0.618 -> 0.096 | Helped |
| use frame bunches instead of a single Is and Im (ablation) | RMB-Net · Synthetic data | PSNR: 19.95 -> 24.06 | Helped |
| use rolling-mixed-bit spikings instead of pure single-bit spikings (ablation) | RMB-Net · Synthetic data | PSNR: 19.37 -> 24.06 | Helped |
| use rolling-mixed-bit spikings instead of pure 3-bit spikings (ablation) | RMB-Net · Synthetic data | PSNR: 26.62 -> 24.06 | Hurt |
| use rolling-mixed-bit spikings instead of full-frame multi-bit spikings (ablation) | RMB-Net · Synthetic data | PSNR: 24.50 -> 24.06 | Hurt |
| add cross-bit attention instead of concatenation (ablation) | RMB-Net · Synthetic data | PSNR: 22.29 -> 24.06 | Helped |
| add cross-time attention instead of temporal average (ablation) | RMB-Net · Synthetic data | PSNR: 21.81 -> 24.06 | Helped |
Surprising findings
- Pure 3-bit input (Pure-M) gives higher PSNR than the full RMB-Net, so the RMB method trades some accuracy for lower data volume.
Limitations
- Real RMB spikings are simulated by 2x2x2 binning because the camera lacks RMB hardware, so there is no ground truth for real data.
- Baselines were designed for single-bit spikings, so the comparison is acknowledged as not fully fair.
- Performance remains below pure multi-bit input (Pure-M) on synthetic data.
Experimental setup
Datasets: synthetic HDR/HFR spiking dataset from alternating-exposure videos real-synthetic spiking data (10 groups)
Benchmarks: synthetic spiking HDR/HFR dataset real-synthetic spiking data
Metrics: PSNR SSIM HDR-VDP-3 HDR-VQM
Baselines: TFW-S TFW-L TFI Spk2ImgNet GC20 MG20
Builds on: Spike2Flow Spk2ImgNet ConvLSTM TFI TFW
Closest papers by meaning
- HDRFlow: Real-Time HDR Video Reconstruction with Large Motions (CVPR 2024 · similarity 0.84)
- EvHDR-GS: Event-guided HDR Video Reconstruction with 3D Gaussian Splatting (AAAI 2025 · similarity 0.84)
- Rethinking High-speed Image Reconstruction Framework with Spike Camera (AAAI 2025 · similarity 0.84)
- Single Shot HDR Recovery via a Video Diffusion Prior (NeurIPS 2026 · similarity 0.83)
- Spatio-Temporal Interactive Learning for Efficient Image Reconstruction of Spiking Cameras (NeurIPS 2024 · similarity 0.83)
- High Dynamic Range Video Compression: A Large-Scale Benchmark Dataset and A Learned Bit-depth Scalable Compression Algorithm (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.