Research map / Continual learning / Online and replay-based
Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SA-SNN, a spiking network for class-incremental continual learning that uses trace-based K-WTA and variable firing thresholds to route tasks through sparse neuron subpopulations, without task labels or replay.
Problem
Catastrophic forgetting in class-incremental continual learning, where the goal is to learn new classes without forgetting old ones while using limited computing resources, explored here with spiking neural networks.
Approach
Combines a temporal trace-based K-Winner-Take-All mask (trace from the STDP-style decaying variable instead of spike counts) with irreversible variable firing thresholds so that different tasks activate different neuron subpopulations in an SNN, plus L2 normalization of inputs, Dale's rule and SGD.
Key results
- SA-SNN reaches 60.06% on splitMNIST and 77.73% on splitCIFAR10 in class-incremental setting, above SDMLP (46.67%, 73.27%) and FlyModel (76.97%, 70.09%) on splitCIFAR10 (FlyModel is higher on splitMNIST).
- SA-SNN + EWC reaches 82.18% on splitMNIST and 80.39% on splitCIFAR10, versus 79.61% and 78.64% for SDMLP + EWC.
- On splitCIFAR100 (25 tasks), SA-SNN reaches 22.86% versus 21.37% for SA-SNN(rate) and 4.28% with no continual learning method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add trace-based K-WTA instead of using a rate-based mask (ablation) | SA-SNN · splitCIFAR10 | validation accuracy: 76.88% -> 77.73% | Helped |
| add trace-based K-WTA instead of using a rate-based mask (ablation) | SA-SNN · splitMNIST | validation accuracy: 50.22 -> 60.06 | Helped |
| add trace-based K-WTA instead of using a rate-based mask (ablation) | SA-SNN · splitCIFAR100 | validation accuracy: 21.37 -> 22.86 | Helped |
| remove variable firing thresholds (ablation) | SA-SNN · splitCIFAR10 | validation accuracy | Hurt |
| remove trace-based K-WTA (ablation) | SA-SNN · splitCIFAR10 | validation accuracy | Hurt |
| remove Dale’s rules (ablation) | SA-SNN · splitCIFAR10 | validation accuracy | Hurt |
| remove input normalization (ablation) | SA-SNN · splitCIFAR10 | validation accuracy | Hurt |
| add EWC regularization | SA-SNN · splitMNIST | validation accuracy: SA-SNN + EWC: 82.18; SDMLP + EWC: 79.61 | Helped |
| add EWC regularization | SA-SNN · splitCIFAR10 | validation accuracy: SA-SNN + EWC: 80.39; SDMLP + EWC: 78.64 | Helped |
| increase the maximum firing threshold (ablation) | SA-SNN · splitCIFAR10 | validation accuracy | No clear effect |
| vary threshold-change rate p (ablation) | SA-SNN · splitCIFAR10 | validation accuracy | Mixed |
Surprising findings
- EWC and MAS applied to SNNs gave only about 30% on splitCIFAR10, far below their gains on ANNs.
- The trace-based K-WTA mask outperformed the rate-based mask, which the authors attribute to tolerating ambiguous outputs and mixed selectivity.
Limitations
- Evaluated only on small benchmarks (MNIST, CIFAR10, CIFAR100) with a single hidden-layer classifier on pretrained embeddings for CIFAR.
- Several hyperparameters (p, tau, K, Thmin, Thmax) are tuned empirically and p is very large (2000000), which may be sensitive.
- Absolute accuracies on splitCIFAR10 remain modest and the paper reports the SNN EWC/MAS baselines as comparatively weak.
Experimental setup
Datasets: MNIST CIFAR10 CIFAR100 NMNIST
Benchmarks: splitMNIST splitCIFAR10 splitCIFAR100
Metrics: test accuracy (class-incremental)
Baselines: None Joint EWC-SNN MAS-SNN SDMLP FlyModel SDMLP + EWC SA-SNN(rate)
Builds on: STBP SDMLP EWC Flymodel (fly hashing K-WTA)
Closest papers by meaning
- Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN (ICLR 2024 · similarity 0.87)
- Improving the Sparse Structure Learning of Spiking Neural Networks from the View of Compression Efficiency (ICLR 2025 · similarity 0.87)
- ALADE-SNN: Adaptive Logit Alignment in Dynamically Expandable Spiking Neural Networks for Class Incremental Learning (AAAI 2025 · similarity 0.87)
- SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models (AAAI 2025 · similarity 0.87)
- Temporal Separation with Entropy Regularization for Knowledge Distillation in Spiking Neural Networks (CVPR 2025 · similarity 0.86)
- Temporal Flexibility in Spiking Neural Networks: Towards Generalization Across Time Steps and Deployment Friendliness (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.