Research map / 3D vision / Depth and stereo
Joint Demosaicing and Denoising for Spike Camera
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SJDD-Net, an iterative unrolled network for joint demosaicing and denoising of Bayer-pattern spike camera streams, plus a spike camera simulator and the first real-world captured Bayer spike stream dataset (BSS).
Problem
Reconstructing clean color images from Bayer-pattern spike camera streams, which are binary, spatially subsampled by the color filter array, and severely noisy (including quantization noise); prior spike demosaicing methods largely ignore noise and there is no real-world training or evaluation data.
Approach
An optimization-derived (half-quadratic splitting) iterative network with three modules: a color spike representation using offset-sharing deformable convolution to align temporal features across color channels, a spike noise estimator producing noise features, and a color correlation prior that fuses per-color encoders across scales.
Key results
- On synthetic DSPK with Gaussian noise sigma=15, the method reaches 32.01 dB PSNR / 0.9077 SSIM, versus 29.27 dB for MPRNet-3DRI and 28.51 dB for the best other baseline shown.
- In the blind Gaussian-Poisson noise setting on DSPK, the method reaches 34.45 dB PSNR versus 34.06 dB for the best baseline (3DRI with FastDVDNet-style pipeline row).
- Ablation on DSPK (GN sigma=15) shows removing offset sharing drops PSNR from 32.01 dB to 29.58 dB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the multi-scale 3D encoder in CSR (ablation) | SJDD-Net · DSPK, GN σ = 15 | PSNR: 30.99dB -> 32.01dB | Helped |
| use offset-sharing deformable convolution instead of independent per-channel deformable convolution (ablation) | SJDD-Net · DSPK, GN σ = 15 | PSNR: 29.58dB -> 32.01dB | Helped |
| include the color mask in offset computation (ablation) | SJDD-Net · DSPK, GN σ = 15 | PSNR: 31.16dB -> 32.01dB | Helped |
| add spike noise estimator features to CCP (ablation) | SJDD-Net · DSPK, GN σ = 15 | PSNR: 30.83dB -> 32.01dB | Helped |
| use the multi-head encoder design in CCP instead of a single-head encoder (ablation) | SJDD-Net · DSPK, GN σ = 15 | PSNR: 31.11dB -> 32.01dB | Helped |
| use SJDD-Net rather than GCPNet-TFI | spike camera JDD · DSPK, GN σ = 0 | PSNR: 34.46 -> 34.72 | Helped |
| use SJDD-Net rather than GCPNet-TFI | spike camera JDD · RSPK, GN σ = 0 | SSIM: 0.9315 -> 0.9256 | Hurt |
| use SJDD-Net rather than MPRNet-3DRI | spike camera JDD · RSPK, GPN | PSNR: 33.99 -> 32.63 | Hurt |
Surprising findings
- GCPNet with raw spike input performs worse than GCPNet with TFI or TFP preprocessing, since binary spike streams differ greatly from raw images.
- The single-frame MPRNet with 3DRI demosaicing outperforms the multi-frame FastDVDNet-3DRI baseline in several settings.
Limitations
- Non-blind evaluation requires known noise levels for some baselines; the method is evaluated mainly on simulated noise.
- Real-world BSS dataset has only 28 sequences and lacks ground-truth clean images, so evaluation there is qualitative.
Experimental setup
Datasets: REDS DAVIS
Benchmarks: DSPK RSPK BSS
Metrics: PSNR SSIM
Baselines: 3DRI with MPRNet 3DRI with FastDVDNet TENet GCPNet GCPNet-TFI GCPNet-TFP
Builds on: 3DRI Spk2ImgNet half-quadratic splitting GCPNet TENet
Closest papers by meaning
- Self-Supervised Learning for Color Spike Camera Reconstruction (CVPR 2025 · similarity 0.92)
- Super-Resolution Reconstruction from Bayer-Pattern Spike Streams (CVPR 2024 · similarity 0.90)
- Spk2SRImgNet: Super-Resolve Dynamic Scene from Spike Stream via Motion Aligned Collaborative Filtering (CVPR 2025 · similarity 0.87)
- SpikeReveal: Unlocking Temporal Sequences from Real Blurry Inputs with Spike Streams (NeurIPS 2024 · similarity 0.87)
- Rethinking High-speed Image Reconstruction Framework with Spike Camera (AAAI 2025 · similarity 0.86)
- Spike-guided Motion Deblurring with Unknown Modal Spatiotemporal Alignment (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.