Research map / Architectures and efficiency / Mixture of experts
Spiking Transformer with Experts Mixture
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Spiking Experts Mixture Mechanism (SEMM), a spike-driven MoE formulation for SNNs, and embeds it into Spiking Transformers as EMSA (head-wise experts in attention) and EMSP (channel-wise experts in MLP), reporting gains on static and neuromorphic datasets.
Problem
Mixture-of-Experts relies on softmax weighting and TopK hard sparsification, which conflict with SNN spike-driven, multiplication-free computation; there was no MoE design for Spiking Transformers.
Approach
Replace softmax/TopK routing with spiking routers whose binary outputs gate spiking experts via element-wise Hadamard product and addition, giving spike-driven dynamic sparse conditional computation with parameter counts comparable to the original SSA and MLP.
Key results
- Spike-driven Transformer-8-512 with SEMM reaches 76.62% top-1 on ImageNet at 4 time steps vs. 74.57% baseline.
- Spikformer-8-512 with SEMM reaches 75.93% vs. 73.38% baseline on ImageNet with fewer parameters (28.22M vs 29.68M).
- SEMM reports CIFAR10 96.16%, CIFAR100 80.26%, CIFAR10-DVS 82.9%, and DVS128 Gesture 99.3% (best-reported configurations).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SEMM to Spikformer-8-512 | Spikformer-8-512 · ImageNet-1k | Top-1 accuracy: 73.38 -> 75.93 | Helped |
| Add SEMM to SD-Transformer-8-512 | SD-Transformer-8-512 · ImageNet-1k | Top-1 accuracy: 74.57 -> 76.62 | Helped |
| Add SEMM to Spikingformer-8-512 | Spikingformer-8-512 · ImageNet-1k | Top-1 accuracy: 74.79 -> 76.03 | Helped |
| Add SEMM to Spikformer | Spikformer · CIFAR10-DVS (T=16) | accuracy: 80.90 -> 82.90 | Helped |
| Add SEMM to Spike-Driven Transformer | Spike-Driven Transformer · DVS128 Gesture (T=16) | accuracy: 99.30 -> 99.30 | No clear effect |
| Add SEMM to Spikingformer | Spikingformer · CIFAR-10 (T=4) | accuracy: 95.81 -> 96.16 | Helped |
| Add SEMM to Spike-Driven Transformer | Spike-Driven Transformer · CIFAR-100 (T=4) | accuracy: 78.40 -> 80.26 | Helped |
| Add depth-wise convolution to EMSP (ablation) | Spikformer EMSP · CIFAR-100 | accuracy: 78.17 -> 78.53 | Helped |
| Add depth-wise convolution to EMSP (ablation) | Spikformer EMSP · CIFAR10-DVS | accuracy: 81.30 -> 82.32 | Helped |
| Add depth-wise convolution to EMSP (ablation) | Spike-driven Transformer EMSP · CIFAR-100 | accuracy: 78.95 -> 79.81 | Helped |
| Add depth-wise convolution to EMSP (ablation) | Spike-driven Transformer EMSP · CIFAR10-DVS | accuracy: 80.51 -> 81.10 | Helped |
| Add depth-wise convolution to EMSP (ablation) | Spikingformer EMSP · CIFAR-100 | accuracy: 79.44 -> 79.81 | Helped |
Surprising findings
- Removing the spiking router drops accuracy so much that Spikingformer with EMSA/EMSP falls below its baseline on CIFAR100.
- EMSP router spike rate decreases strongly with depth (0.25 in block 0 to 0.02 in block 7), indicating highly sparse deep-layer routing.
Limitations
- No measured energy or hardware deployment results; hardware feasibility is only discussed theoretically.
- No error bars or statistical significance reported.
- Evaluated only on classification tasks and relatively small or mid-size Spiking Transformer variants.
Experimental setup
Datasets: ImageNet-1k CIFAR10 CIFAR100 CIFAR10-DVS DVS128 Gesture
Benchmarks: ImageNet-1k CIFAR10 CIFAR100 CIFAR10-DVS DVS128 Gesture
Metrics: top-1 accuracy average spiking rate
Baselines: Spikformer Spike-driven Transformer Spikingformer SEW-ResNet MS-ResNet tdBN PLIF DIET-SNN Dspike DSR ANN-MoE with heavy MLP experts
Builds on: Spikformer Spike-driven Transformer Spikingformer Sparsely-gated mixture-of-experts SEW ResNet
Closest papers by meaning
- Improving the Sparse Structure Learning of Spiking Neural Networks from the View of Compression Efficiency (ICLR 2025 · similarity 0.87)
- EnOF-SNN: Training Accurate Spiking Neural Networks via Enhancing the Output Feature (NeurIPS 2024 · similarity 0.87)
- Synergy Between the Strong and the Weak: Spiking Neural Networks are Inherently Self-Distillers (NeurIPS 2025 · similarity 0.86)
- Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion (NeurIPS 2024 · similarity 0.86)
- SpikedAttention: Training-Free and Fully Spike-Driven Transformer-to-SNN Conversion with Winner-Oriented Spike Shift for Softmax Operation (NeurIPS 2024 · similarity 0.86)
- Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.