Research map / Architectures and efficiency / Spiking networks
Spiking Neural Networks Need High-Frequency Information
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Argues that spiking neurons act as low-pass filters that suppress high-frequency information in SNNs, and proposes Max-Former (Spiking Transformer with extra Max-Pool in patch embedding and depth-wise convolution replacing early self-attention) plus Max-ResNet, reaching 82.39% on ImageNet and 97.17% on CIFAR-10 with direct training.
Problem
Spiking neural networks lag behind ANNs in accuracy; the common explanation (binary spike representation error) is questioned, and the paper asks whether frequency properties of spiking neurons limit feature representation.
Approach
Spiking neurons' membrane dynamics form a first-order IIR low-pass filter, so SNNs propagate low-frequency information; restoring high-frequency signals via Max-Pool (high-pass) in patch embedding and depth-wise convolution in early stages improves Spiking Transformers and ResNets.
Key results
- Max-Former-10-768 (T=4) reaches 82.39% top-1 on ImageNet with 63.99M params and 14.87 mJ, versus Spikformer 74.81% (66.34M).
- Max-Former achieves 97.04% on CIFAR-10 and 82.65% on CIFAR-100 at T=4 with 6.57M params, outperforming MS-QKFormer (96.84%, 81.57%) under identical training.
- Max-ResNet-18 (direct training) reaches 97.17% on CIFAR-10 and 83.06% on CIFAR-100, improving MS-ResNet-18 by +2.25% and +6.65%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Avg-Pool with Max-Pool across all token-mixing stages (ablation) | Spiking Transformer with Embed-Orig patch embedding · CIFAR-100 | accuracy: 76.73% to 79.12% | Helped |
| add Max-Pool in the middle patch-embedding block (ablation) | Spiking Transformer · CIFAR-100 | accuracy: 80.11% to 80.46% | Helped |
| replace Embed-Orig/Embed-Max/Embed-Max with Embed-Orig/Embed-Orig/Embed-Orig (ablation) | Max-Former · CIFAR-100 | accuracy: 82.65% to 81.63% | Hurt |
| replace first-stage Embed-Max+ with Embed-Orig (ablation) | Max-Former · CIFAR10-DVS | accuracy: 84.2% to 81.5% | Hurt |
| replace early-stage DWC token mixing with SSA (ablation) | Max-Former · CIFAR-100 | accuracy: 82.23% to 82.65% | Hurt |
| replace DWC-3 + SSA hybrid token mixing with full SSA (ablation) | Max-Former · CIFAR10-DVS | accuracy: 83.9% to 84.2% | Hurt |
| use larger DWC-5/DWC-7 kernels instead of the DWC-3 configuration (ablation) | Max-Former · CIFAR-100 | accuracy: -0.63% | Hurt |
| use DWC-5/DWC-7 instead of DWC-3 (ablation) | Max-Former · CIFAR10-DVS | accuracy: -2.1% | Hurt |
| use DWC-1 instead of DWC-3 for token mixing (ablation) | Max-Former · CIFAR10-DVS | accuracy: 81.2% vs 84.2% | Hurt |
| add two max-pooling operations relative to MS-ResNet | MS-ResNet-18 [3, 3, 2] · CIFAR-10 | accuracy: +2.25% (94.92% to 97.17%) | Helped |
| add two max-pooling operations relative to MS-ResNet | MS-ResNet-18 [3, 3, 2] · CIFAR-100 | accuracy: +6.65% (76.41% to 83.06%) | Helped |
Surprising findings
- Replacing Avg-Pool with Max-Pool for token mixing in Spiking Transformers improves CIFAR-100 accuracy from 76.73% to 79.12%, opposite to the ANN preference for Avg-Pool.
- Spiking outputs appear spectrally all-pass due to impulse-shaped waveforms, yet the network-level propagation is low-pass.
Limitations
- Max-Former requires manually balancing high- and low-frequency components (kernel sizes, placement of Max-Pool), needing task-specific expertise.
- No error bars reported due to computational cost.
- Energy figures are theoretical SOP-based estimates rather than measured on neuromorphic hardware.
Experimental setup
Datasets: ImageNet-1k CIFAR-10 CIFAR-100 CIFAR10-DVS DVS128 Gesture
Benchmarks: ImageNet-1k CIFAR-10 CIFAR-100 CIFAR10-DVS DVS128 Gesture
Metrics: top-1 accuracy parameters (M) energy (mJ) training/inference time and memory
Baselines: Spikformer S-Transformer SWformer QKFormer MS-QKFormer MS-ResNet-18 KDSNN-ResNet-18 MST Meta-Spikformer PVT-Large DeiT-B ViT-L/16
Builds on: Spikformer QKFormer MS-ResNet Spikformer v2 SpikingJelly
Closest papers by meaning
- Balanced Resonate-and-Fire Neurons (ICML 2024 · similarity 0.89)
- FSTA-SNN:Frequency-Based Spatial-Temporal Attention Module for Spiking Neural Networks (AAAI 2025 · similarity 0.88)
- FiTS: Interpretable Spiking Neurons via Frequency Selectivity and Temporal Shaping (NeurIPS 2026 · similarity 0.88)
- Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers (NeurIPS 2025 · similarity 0.88)
- Rethinking the Membrane Dynamics and Optimization Objectives of Spiking Neural Networks (NeurIPS 2024 · similarity 0.88)
- A Scalable, Causal, and Energy Efficient Framework for Neural Decoding with Spiking Neural Networks (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.