Research map / Architectures and efficiency / Spiking networks
Toward Relative Positional Encoding in Spiking Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Gray-PE and Log-PE, two relative positional encodings for spiking Transformers that keep spike (binary) representations, using XNOR-based spiking self-attention, and evaluates them on time-series forecasting, text classification and patch-based image classification.
Problem
Relative positional encoding has not been integrated into spiking Transformers, since adding RPE (ALiBi, RoPE) to binary spike queries and keys disrupts spike representations and performance.
Approach
Gray Code encodings of position indices concatenated to binary Q/K, where a power-of-two relative offset yields a constant Hamming distance (proved), combined with XNOR-based attention; Log-PE adds a non-negative logarithmic relative distance bias directly to the integer attention map.
Key results
- Spikformer-XNOR with Log-PE reaches avg R^2 0.750 on time-series forecasting versus 0.733 for Spikformer with Conv-PE (Original) and 0.744 for CPG-PE.
- On text classification, Spikformer-XNOR with Log-PE achieves 80.38% average accuracy versus 77.25% for Spikformer without PE and 79.41% with CPG-PE.
- On CIFAR10-DVS, Spikformer-XNOR with Gray-PE 2D reaches 78.70% and average accuracy 75.39% versus 74.51% for the original Spikformer with Conv-PE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Log-PE with XNOR attention | Spikformer · Time-series forecasting (4 benchmarks, Avg.) | R²: .733 -> .750 | Helped |
| Add Gray-PE with XNOR attention | Spikformer · Text classification (6 benchmarks, Avg.) | accuracy: 77.25 -> 80.11 | Helped |
| Add Log-PE with XNOR attention | Spikformer · Long text classification (Avg.) | accuracy: 81.46 -> 83.62 | Helped |
| Use 2D rather than 1D Gray-PE (ablation) | Spikformer-XNOR · Patch-based image classification (Avg.) | accuracy: 74.95 -> 75.39; 0.44% | Helped |
| Replace dot-product attention with XNOR attention while retaining Conv-PE (ablation) | Spikformer · Time-series forecasting (4 benchmarks, Avg.) | R²: .733 -> .733 | No clear effect |
| Remove XNOR attention while retaining Log-PE (ablation) | Spikformer-XNOR w/ Log-PE · Electricity (prediction length 24) | R²: .974 -> .710* | Hurt |
| Replace Log-PE with complete RPE (ablation) | Spikformer-XNOR · Metr-la (prediction length 24) | R²: .535 -> .158* | Hurt |
| Replace Log-PE with complete RPE (ablation) | Spikformer-XNOR · Electricity (prediction length 24) | R²: .974 -> .710* | Hurt |
| Add Log-PE rather than Gray-PE | Spikformer-XNOR · Long text classification (Avg.) | accuracy: 82.36 -> 83.62 | Helped |
| Add Gray-PE or Log-PE | QKFormer · Patch-based image classification | accuracy: 79.67 -> 79.03; 79.67 -> 79.56 | Mixed |
| Use Gray-PE 2D rather than Gray-PE 1D (ablation) | QKFormer-XNOR · Patch-based image classification (Avg.) | accuracy: 79.03 -> 79.56 | Helped |
| Use 2D Log-PE (ablation) | Spiking Transformer · Patch-based image classification | convergence | Hurt |
Surprising findings
- Directly applying ALiBi or RoPE to Spikformer yields poor results, and for ALiBi and RoPE the R^2 collapses on long-horizon Pems-bay and Solar settings.
- QKFormer shows insensitivity to positional encoding in image classification, with minimal gains or degradation from CPG-PE and Gray-PE.
- Log-2D positional encoding failed to converge in pre-experiments and was abandoned.
Limitations
- Scalability concerns for extremely long sequences (e.g., length 10240).
- Gray-PE and Log-PE may limit flexibility across data modalities and tasks.
- Evaluation limited to time series, text, and patch-based image classification; QKFormer shows little sensitivity to PE in image tasks.
Experimental setup
Datasets: Metr-la Pems-bay Solar Electricity MR SST-2 SST-5 Subj ChnSenti Waimai AGNEWS IMDB
Benchmarks: Metr-la Pems-bay Solar Electricity MR SST-2 SST-5 Subj ChnSenti Waimai AGNEWS IMDB
Metrics: R^2 RSE accuracy
Baselines: Spikformer w/ Conv-PE CPG-PE ALiBi RoPE Sin-PE Spikingformer SDT-V1 QKFormer Fine-tuned BERT Vision-Transformer
Builds on: Spikformer ALiBi RoPE SeqSNN QKFormer
Closest papers by meaning
- Advancing Spiking Neural Networks for Sequential Modeling with Central Pattern Generators (NeurIPS 2024 · similarity 0.91)
- Spiking Transformer: Introducing Accurate Addition-Only Spiking Self-Attention for Transformer (CVPR 2025 · similarity 0.89)
- STEP: A Unified Spiking Transformer Evaluation Platform for Fair and Reproducible Benchmarking (NeurIPS 2025 · similarity 0.89)
- Towards Efficient Spiking Transformer: a Token Sparsification Framework for Training and Inference Acceleration (ICML 2024 · similarity 0.88)
- STAA-SNN: Spatial-Temporal Attention Aggregator for Spiking Neural Networks (CVPR 2025 · similarity 0.88)
- Spiking Transformer with Spatial-Temporal Attention (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.