Research map / Image generation / Image restoration
Rethinking High-speed Image Reconstruction Framework with Spike Camera
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpikeCLIP, a three-stage CLIP-guided framework that reconstructs images from low-light spike streams using TFI coarse supervision, learnable HQ/LQ prompts, and class-label InfoNCE loss, without paired ground truth.
Problem
Spike-camera image reconstruction under low-light conditions degrades for supervised models trained on synthetic data (domain gap) and for self-supervised methods relying on poor TFI pseudo-labels.
Approach
Replaces ground-truth sharp-image supervision with CLIP-based signals: class-label text features (InfoNCE) and learnable prompts trained to distinguish unpaired high-quality images from low-quality reconstructions, using a lightweight network (LRN) with TFI coarse initialization.
Key results
- On U-CALTECH, SpikeCLIP achieves NIQE 4.657, BRISQUE 14.955, PIQE 3.265, versus best baseline WGSE NIQE 7.449.
- On U-CIFAR, SpikeCLIP achieves NIQE 5.383, BRISQUE 25.831, PIQE 4.298 with 0.186M parameters and 0.540 ms latency.
- Ablation: adding L_class to TFI-initialized network raises classification ACC from 1.60% to 63.00%; adding prompt loss gives ACC 64.00%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add class-loss training to TFI coarse reconstruction (ablation) | SpikeCLIP · U-CALTECH | accuracy: 1.60 -> 63.00 | Helped |
| add class-loss training to TFI coarse reconstruction (ablation) | SpikeCLIP · U-CALTECH | NIQE: 15.389 -> 4.799 | Helped |
| add TFI coarse reconstruction to class-loss training (ablation) | SpikeCLIP · U-CALTECH | NIQE: 6.043 -> 4.799 | Helped |
| add TFI coarse reconstruction to class-loss training (ablation) | SpikeCLIP · U-CALTECH | accuracy: 58.90 -> 63.00 | Helped |
| add learned HQ prompt loss to TFI and class-loss training (ablation) | SpikeCLIP · U-CALTECH | BRISQUE: 18.748 -> 14.955 | Helped |
| add learned HQ prompt loss to TFI and class-loss training (ablation) | SpikeCLIP · U-CALTECH | accuracy: 63.00 -> 64.00 | Helped |
| replace UNet with lightweight LRN (ablation) | SpikeCLIP · U-CALTECH | NIQE: 25.94 -> 4.657 | Helped |
| replace UNet with lightweight LRN (ablation) | SpikeCLIP · U-CALTECH | accuracy: 61.70 -> 64.00 | Helped |
| use SpikeCLIP instead of WGSE | spike-based reconstruction · U-CALTECH | NIQE: 7.449 -> 4.657 | Helped |
| use SpikeCLIP instead of TFSTP | spike-based reconstruction · U-CALTECH | BRISQUE: 23.446 -> 14.955 | Helped |
| use SpikeCLIP instead of TFI | spike-based reconstruction · U-CALTECH | PIQE: 4.762 -> 3.265 | Helped |
| use SpikeCLIP instead of WGSE | spike-based reconstruction · U-CIFAR | NIQE: 9.458 -> 5.383 | Helped |
Surprising findings
- The TFI-only coarse reconstruction yields only 1.60% classification accuracy, indicating raw TFI features are not aligned with CLIP text space.
Limitations
- Evaluation relies on non-reference metrics (NIQE, BRISQUE, PIQE) because paired real-world ground truth is unavailable.
- Only two low-light datasets (U-CALTECH and U-CIFAR) are used for evaluation, both at 250x250 resolution.
Experimental setup
Datasets: U-CALTECH U-CIFAR Spike-REDS RecVidarReal2019 MotVidarReal2020 MomVidarReal2021 UHSR
Benchmarks: U-CALTECH U-CIFAR
Metrics: NIQE BRISQUE PIQE ACC Params FLOPs Latency
Baselines: TFP TFI TFSTP SSML SpikeFormer RSIR Spk2ImgNet WGSE UNet
Builds on: CLIP CoOp CLIP-LIT TFI WGSE
Closest papers by meaning
- SpikeReveal: Unlocking Temporal Sequences from Real Blurry Inputs with Spike Streams (NeurIPS 2024 · similarity 0.89)
- Spatio-Temporal Interactive Learning for Efficient Image Reconstruction of Spiking Cameras (NeurIPS 2024 · similarity 0.89)
- SpikeNeRF: Learning Neural Radiance Fields from Continuous Spike Stream (CVPR 2024 · similarity 0.89)
- Boosting Spike Camera Image Reconstruction from a Perspective of Dealing with Spike Fluctuations (CVPR 2024 · similarity 0.88)
- Self-Supervised Learning for Color Spike Camera Reconstruction (CVPR 2025 · similarity 0.88)
- Spike-guided Motion Deblurring with Unknown Modal Spatiotemporal Alignment (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.