Research map / Architectures and efficiency / Spiking networks
Finding Visual Saliency in Continuous Spike Stream
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the first visual saliency detection method for continuous spike camera streams, a fully spiking Recurrent Spiking Transformer (RST), plus a new real-world spike-based saliency (SVS) dataset.
Problem
Detecting visual saliency from the continuous, binary, high-rate (20,000 Hz) spike stream of a spike camera while keeping power consumption low, which existing saliency methods (designed for RGB frames) do not address.
Approach
A full-SNN transformer with a recurrent feature aggregation block that uses the next time step's features (reverse recurrence) as keys/values to enhance the current step, keeping all inter-module communication binary via OR-based residuals, trained with a per-step weighted multi-step BCE+IoU+SSIM loss.
Key results
- On the SVS dataset in single-step mode, the proposed RST reaches MAE 0.0784 and max F-beta 0.6313, versus 0.1026 / 0.5310 for Spiking Deeplab, the best prior SNN baseline.
- In multi-step mode, RST reaches MAE 0.0554, max F-beta 0.6981 and S-measure 0.7591, outperforming Spikformer-ADD (MAE 0.0717, S-m 0.7563).
- Reported energy for single-step inference is 5.8 mJ versus 167 mJ for an ANN counterpart, a roughly 28.7x reduction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reverse recurrent mode, fusing step t+1 features into the current features (ablation) | RST · SVS, multi-step | MAE: 0.0581 -> 0.0554 | Helped |
| Use forward recurrent mode, using step t-1 as key and value (ablation) | RST · SVS, multi-step | MAE: 0.0581 -> 0.0611 | Hurt |
| Remove the refinement block (ablation) | RST · SVS, multi-step | MAE: 0.0554 -> 0.0701 | Hurt |
| Vary the number of RFA modules from 6 to 0 (ablation) | RST · SVS, multi-step | MAE: 0.0554 -> 0.0642 | Hurt |
| Vary the number of RFA modules from 6 to 2 (ablation) | RST · SVS, multi-step | MAE: 0.0554 -> 0.0571 | Hurt |
| Vary the number of RFA modules from 6 to 4 (ablation) | RST · SVS, multi-step | MAE: 0.0554 -> 0.0559 | Hurt |
| Vary the number of RFA modules from 6 to 8 (ablation) | RST · SVS, multi-step | MAE: 0.0554 -> 0.0580 | Hurt |
| Apply multi-step loss instead of vanilla loss (ablation) | RST with reverse recurrent mode · SVS, multi-step | MAE: 0.0613 -> 0.0554 | Helped |
| Replace OR with element-wise ADD (ablation) | RST · SVS, multi-step | MAE: 0.0554 -> 0.0576 | Hurt |
| Replace OR with concatenation (ablation) | RST · SVS, multi-step | MAE: 0.0554 -> 0.0567 | Hurt |
| Replace OR with element-wise AND (ablation) | RST · SVS, multi-step | training feasibility | Hurt |
| Use the proposed RST instead of the strongest SNN baseline | SVS, single-step · SVS, single-step | MAE: 0.1026 -> 0.0784 | Helped |
Surprising findings
- Forward recurrence (using the previous step's features) performs worse than the vanilla SNN without recurrence, which the authors attribute to sparse early-step information.
- Replacing the OR residual with element-wise AND causes all spikes to vanish during training, making the model hard to train.
Limitations
- Comparisons are only against SNN baselines adapted from other tasks, since no prior SNN saliency method exists.
- Performance does not scale linearly with the number of RST/RFA modules, attributed to limited dataset size.
Experimental setup
Datasets: SVS (spike-based visual saliency) dataset
Benchmarks: SVS
Metrics: MAE max F-measure mean F-measure structure-measure
Baselines: Spiking Deeplab Spiking FCN EVSNN Spikformer-ADD Spikformer-OR
Builds on: Spikformer Spiking DeepLab EVSNN ConvLSTM
Closest papers by meaning
- SpikeVideoFormer: An Efficient Spike-Driven Video Transformer with Hamming Attention and $\mathcal{O}(T)$ Complexity (ICML 2025 · similarity 0.88)
- SpikeNeRF: Learning Neural Radiance Fields from Continuous Spike Stream (CVPR 2024 · similarity 0.88)
- Rethinking High-speed Image Reconstruction Framework with Spike Camera (AAAI 2025 · similarity 0.88)
- Spiking Vision Transformer with Saccadic Attention (ICLR 2025 · similarity 0.88)
- Spatio-Temporal Interactive Learning for Efficient Image Reconstruction of Spiking Cameras (NeurIPS 2024 · similarity 0.87)
- SpikeReveal: Unlocking Temporal Sequences from Real Blurry Inputs with Spike Streams (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.