Research map / Video / Video segmentation and tracking
Spatio-Temporal Interactive Learning for Efficient Image Reconstruction of Spiking Cameras
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STIR, a coarse-to-fine spatio-temporal interactive network with a hybrid spike embedding (TFP + ResNet) that jointly performs motion estimation and intensity reconstruction from spiking camera streams, achieving higher PSNR and lower cost than prior SOTA.
Problem
Reconstructing intermediate intensity frames from binary spike streams of spiking cameras is typically done with step-by-step pipelines (motion estimation, then intensity recovery) that hinder spatio-temporal interaction and make inference slow and heavy.
Approach
A single encoder-decoder jointly performs warping-based inter-frame feature alignment and synthesis-based intra-frame feature filtering in a coarse-to-fine manner, augmented by a symmetric interactive cross-attention block at the top pyramid level, a multi-motion field estimation block at the bottom level, and a hybrid spike embedding combining TFP with a learned ResNet representation.
Key results
- On the synthetic SREDS dataset, STIR reaches 38.79 dB PSNR and 0.966 SSIM, outperforming WGSE (37.44 dB) and the other baselines.
- STIR has 5.11M parameters and 0.42 TFLOPs, and runs about 11x faster than Spk2ImgNet and 5x faster than WGSE on average.
- Ablations show removing the synthesis-based intra-frame filtering collapses PSNR to 12.19 dB, and the symmetric interactive attention and multi-motion fields each contribute gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STIR instead of the strongest listed baseline, WGSE | Spike-to-image reconstruction · SREDS | PSNR: 37.44 -> 38.79 | Helped |
| Increase the feature pyramid from 3 to 5 levels (ablation) | STIR · SREDS | PSNR: 37.99 -> 38.79 | Helped |
| Add warping-based inter-frame feature alignment (ablation) | STIR motion-intensity interaction · SREDS | PSNR: 37.74 -> 38.79 | Helped |
| Add synthesis-based intra-frame feature filtering (ablation) | STIR motion-intensity interaction · SREDS | PSNR: 12.19 -> 38.79 | Helped |
| Add symmetric interactive attention instead of removing it or using independent cross-attention (ablation) | STIR · SREDS | PSNR: 38.09 / 38.23 -> 38.79 | Helped |
| Use 3 motion-field groups instead of 1 or 5 (ablation) | STIR · SREDS | PSNR: 37.89 -> 38.79; 38.37 -> 38.79 | Mixed |
| Increase feature width from ×0.5 to ×1.0 (ablation) | STIR · SREDS | PSNR: 38.09 -> 38.79 | Helped |
| Add reconstruction loss Lrec (ablation) | STIR training objective · SREDS | PSNR: 36.43 -> 38.79 | Helped |
| Add perceptual loss Lper (ablation) | STIR training objective · SREDS | PSNR: 37.30 -> 38.79 | Helped |
| Add multi-scale consistency loss Lmsc (ablation) | STIR training objective · SREDS | PSNR: 38.54 -> 38.79 | Helped |
| Combine TFP and ResNet as HSER instead of using either alone (ablation) | STIR spike embedding · SREDS | PSNR: 36.39 / 38.01 -> 38.79 | Helped |
| Combine TFP with multi-dilated embedding (ablation) | STIR spike embedding · SREDS | PSNR: 36.39 -> 37.60; 38.79 | Hurt |
Surprising findings
- Removing synthesis-based intra-frame feature filtering causes catastrophic failure (PSNR drops to 12.19 dB).
- Using 5 motion-field groups performed worse than 3 (38.37 vs 38.79 dB), so more motion fields do not monotonically help.
Limitations
- Assumes sufficient illumination; in extremely low light, reconstructions become darker and noisier.
- No error bars or statistical significance reported.
- Evaluated only on spike streams with fixed temporal length N=20.
Experimental setup
Datasets: SREDS momVidarReal2021 recVidarReal2019
Benchmarks: SREDS momVidarReal2021 recVidarReal2019
Metrics: PSNR SSIM LPIPS NIQE BRISQUE Params FLOPs GPU memory runtime
Baselines: TFP TFI TFSTP SSIR ET-Net HyperE2VID Spk2ImgNet WGSE
Builds on: TFP Spk2ImgNet HyperE2VID event-based video frame interpolation with warping and synthesis
Closest papers by meaning
- Spk2SRImgNet: Super-Resolve Dynamic Scene from Spike Stream via Motion Aligned Collaborative Filtering (CVPR 2025 · similarity 0.90)
- Rethinking High-speed Image Reconstruction Framework with Spike Camera (AAAI 2025 · similarity 0.89)
- Spike-guided Motion Deblurring with Unknown Modal Spatiotemporal Alignment (CVPR 2024 · similarity 0.87)
- Fully Spiking Neural Networks for Unified Frame-Event Object Tracking (NeurIPS 2025 · similarity 0.87)
- Leveraging Asynchronous Spiking Neural Networks for Ultra Efficient Event-Based Visual Processing (AAAI 2025 · similarity 0.87)
- USP-Gaussian: Unifying Spike-based Image Reconstruction, Pose Correction and Gaussian Splatting (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.