Research map / Architectures and efficiency / Spiking networks
Rethinking Spiking Self-Attention Mechanism: Implementing a-XNOR Similarity Calculation in Spiking Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes α-XNOR similarity, a spike-aware replacement for dot-product similarity in spiking self-attention, yielding α-SSA modules that improve several spiking Transformers on ImageNet, CIFAR10/100 and CIFAR10-DVS.
Problem
Dot-product similarity between sparse binary spike queries and keys ignores matching non-spike pairs (0-0), so it measures similarity poorly, contributing to the performance gap between spiking Transformers and ANN Transformers.
Approach
Replace the dot product with α-XNOR similarity, which scores matching spikes as 1, matching non-spikes as a weight α in (0,1), and mismatches as 0; a learned-free linear scale and bias on the scores, set analytically to normalize membrane potentials, complements it.
Key results
- α-SSA-Swin-10-512 reaches 80.62% top-1 on ImageNet, versus 79.70% for Meta-SpikeFormer-512 and 75.39% for α-SSA-ViT-8-512 vs 73.38% Spikformer-8-512.
- On CIFAR100 with Spikformer, α-SSA (α=0.3) improves accuracy from 78.21% to 80.22%, with ablation showing α=0.3 beats XNOR (α=1, 79.33%) and dot-product-based linear scaling (α=0, 79.04%).
- Theoretical analysis shows α-XNOR similarity score entropy of 5.50 vs 2.07 for dot product and 3.44 for XNOR in the stated example (fq=fk=0.2, d=32, α=0.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace dot-product SSA with α-SSA (α = 0.3) in S-ViT | S-ViT · ImageNet | accuracy | Helped |
| Replace dot-product SSA with α-SSA (α = 0.3) in M-ViT | M-ViT · ImageNet | accuracy | Helped |
| Replace SSA with α-SSA (α = 0.3) in S-ViT | S-ViT · CIFAR10 | accuracy | Helped |
| Replace SSA with α-SSA (α = 0.3) in M-ViT | M-ViT · CIFAR100 | accuracy | Helped |
| Replace SSA with α-SSA (α = 0.3) in M-ViT | M-ViT · CIFAR10-DVS | accuracy | Helped |
| Replace SSA with α-SSA (α = 0.5) in M-ViT | M-ViT · CIFAR10-DVS | accuracy | Helped |
| Add the α-SSA attention-score linear transformation while retaining dot-product similarity (α = 0) (ablation) | Spikformer · CIFAR100 | accuracy: 78.21% to 79.04% | Helped |
| Replace dot-product similarity with XNOR similarity (α = 1) (ablation) | Spikformer + α-SSA · CIFAR100 | accuracy: 79.04% to 79.33% | Helped |
| Replace dot-product similarity with α-XNOR similarity (α = 0.3) (ablation) | Spikformer + α-SSA · CIFAR100 | accuracy: 2.01% | Helped |
| Use α = 0.3 instead of α = 0.5 in S-ViT (ablation) | S-ViT + α-SSA · CIFAR10 | accuracy: 95.92 to 95.97 | Helped |
| Use α = 0.3 instead of α = 0.5 in S-ViT (ablation) | S-ViT + α-SSA · CIFAR100 | accuracy: 79.97 to 80.22 | Helped |
| Use α = 0.3 instead of α = 0.5 in M-ViT (ablation) | M-ViT + α-SSA · CIFAR10-DVS | accuracy: 82.24 to 82.22 | No clear effect |
Surprising findings
- Performance gains are smaller for multi-stage M-ViT models (Swin-style) than single-stage S-ViT models, attributed to the minor role of self-attention in such designs.
Limitations
- Performance gains are smaller for multi-stage architectures where self-attention plays a minor role.
- α is a hand-tuned hyperparameter (best range 0.2 to 0.4), with no adaptive selection.
Experimental setup
Datasets: ImageNet CIFAR10 CIFAR100 CIFAR10-DVS
Benchmarks: ImageNet CIFAR10 CIFAR100 CIFAR10-DVS
Metrics: top-1 accuracy parameter count information entropy
Baselines: Spikformer Spikingformer Spike-driven Transformer SpikingResformer Meta-SpikeFormer SEW ResNet MS-ResNet TET DSpike
Builds on: Spikformer Spike-driven Transformer Meta-SpikeFormer Spikingformer Leaky Integrate-and-Fire
Closest papers by meaning
- STEP: A Unified Spiking Transformer Evaluation Platform for Fair and Reproducible Benchmarking (NeurIPS 2025 · similarity 0.89)
- Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer (CVPR 2025 · similarity 0.89)
- Towards Efficient Spiking Transformer: a Token Sparsification Framework for Training and Inference Acceleration (ICML 2024 · similarity 0.87)
- Toward Relative Positional Encoding in Spiking Transformers (NeurIPS 2025 · similarity 0.86)
- SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks (CVPR 2024 · similarity 0.86)
- Spiking Transformer with Spatial-Temporal Attention (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.