Research map / 3D vision / Depth and stereo
Self-Supervised Learning for Color Spike Camera Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a motion-guided preliminary reconstruction plus a self-supervised masked-spike-encoder network that restores color images from Bayer-pattern color spike camera streams.
Problem
Restoring high-quality color images from binary Bayer-pattern spike streams of color spike cameras, which face missing per-channel pixels, high-speed motion blur, and quantization noise, without ground-truth supervision for real data.
Approach
Fill missing color-channel pixels by collecting same-color temporal pixels along motion trajectories estimated jointly from all color channels, then train a self-supervised network with these preliminary results as pseudo-labels, using masked 3D partial convolution and temporal masked convolution to suppress temporally correlated quantization noise.
Key results
- On RBSS at alpha=0.6, the proposed self-supervised method reaches 32.90 dB PSNR, the best among methods without ground truth.
- On GBSS, the method achieves 35.73 dB PSNR, above the self-supervised and learning-free baselines, though below supervised SJDD-Net and CSpkNet (about 38 dB).
- Visual comparisons on real-world BSS data show less noise and better detail than supervised SJDD-Net and CSpkNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove joint motion estimation (ablation) | Motion-guided reconstruction · RBSS | PSNR: 31.42, 29.65, 28.08 -> 31.27, 29.50, 27.91 | Hurt |
| remove spatial interpolation (ablation) | Motion-guided reconstruction · RBSS | PSNR: 31.42, 29.65, 28.08 -> 31.44, 29.51, 27.81 | Hurt |
| remove motion-guided search (ablation) | Motion-guided reconstruction · RBSS | PSNR: 31.42, 29.65, 28.08 -> 28.48, 25.96, 24.17 | Hurt |
| replace TFI with TFP inference (short, medium, or long window) (ablation) | Motion-guided reconstruction · RBSS | PSNR: TFP short: 30.25, 28.84, 27.41; medium: 30.07, 27.99, 26.21; long: 28.98, 26.71, 24.87 | Hurt |
| remove color-layout constraint from pixel search (ablation) | Motion-guided reconstruction · RBSS | PSNR: 31.42, 29.65, 28.08 -> 30.16, 27.86, 26.23 | Hurt |
| remove filling of positions without temporal pixels (ablation) | Motion-guided reconstruction · RBSS | PSNR: 31.42, 29.65, 28.08 -> 31.31, 29.54, 27.96 | Hurt |
| replace Oursmg pseudo-labels with TFI pseudo-labels (ablation) | Self-supervised reconstruction network · RBSS | PSNR: 30.61, 28.76, 27.06 | Hurt |
| replace Oursmg pseudo-labels with TFP pseudo-labels (ablation) | Self-supervised reconstruction network · RBSS | PSNR: short: 28.05, 26.34, 24.60; medium: 28.65, 26.82, 25.14; long: 28.87, 26.50, 24.60 | Hurt |
| replace masked spike encoder with 2D convolution (ablation) | Self-supervised reconstruction network · RBSS | PSNR: 32.90, 31.34, 29.95 -> 30.18, 28.22, 26.47 | Hurt |
| replace masked spike encoder with 3D convolution (ablation) | Self-supervised reconstruction network · RBSS | PSNR: 32.90, 31.34, 29.95 -> 30.31, 28.36, 26.66 | Hurt |
| remove MPConv from masked spike encoder (ablation) | Self-supervised reconstruction network · RBSS | PSNR: 32.90, 31.34, 29.95 -> 30.60, 28.72, 27.07 | Hurt |
| use only one 3DConv or one TMConv in the encoder (ablation) | Self-supervised reconstruction network · RBSS | PSNR: single 3DConv: 32.81, 31.25, 29.84; single TMConv: 32.83, 31.27, 29.86 | Hurt |
Surprising findings
- The self-supervised method, trained without ground truth, is reported to outperform supervised SOTA methods on real-world captured data, while lagging them on synthetic data.
Limitations
- Does not exceed supervised methods on synthetic data with ground truth.
- Quantization noise handling relies on a noise analysis from a simulator, which may not match real hardware exactly.
- Motion-guided preliminary reconstruction is sensitive to motion estimation quality under high-speed motion.
Experimental setup
Datasets: REDS DAVIS GoPro BSS
Benchmarks: RBSS DBSS GBSS BSS
Metrics: PSNR SSIM LPIPS FLOPs Params running time
Baselines: SJDD-Net CSpkNet WGSE 3DRI TFI TFP STP SJRE
Builds on: Spike2Flow TFI partial convolution blind-spot denoising SJRE
Closest papers by meaning
- Joint Demosaicing and Denoising for Spike Camera (AAAI 2024 · similarity 0.92)
- Super-Resolution Reconstruction from Bayer-Pattern Spike Streams (CVPR 2024 · similarity 0.92)
- SpikeReveal: Unlocking Temporal Sequences from Real Blurry Inputs with Spike Streams (NeurIPS 2024 · similarity 0.88)
- Rethinking High-speed Image Reconstruction Framework with Spike Camera (AAAI 2025 · similarity 0.88)
- Spatio-Temporal Interactive Learning for Efficient Image Reconstruction of Spiking Cameras (NeurIPS 2024 · similarity 0.86)
- Spk2SRImgNet: Super-Resolve Dynamic Scene from Spike Stream via Motion Aligned Collaborative Filtering (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.