Research map / Large language models / Language, safety and interpretability
SpikeLLM: Scaling up Spiking Neural Network to Large Language Models via Saliency-based Spiking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpikeLLM, the first spiking LLM scaling 7–70B parameter models via Generalized Integrate-and-Fire neurons and saliency-based Optimal Brain Spiking, built on OmniQuant and GPTQ post-training pipelines.
Problem
Spiking neural networks have been limited to under 1B parameters because firing-rate encoding is inefficient and direct training or ANN-SNN conversion does not scale to LLMs; LLM quantization also suffers errors on outlier/salient channels.
Approach
Generalized Integrate-and-Fire (GIF) neurons merge L IF steps so each step emits log2 L bits, and Optimal Brain Spiking uses first-order activation and second-order weight saliency to give salient channels more spiking steps (about log2 T bits on average), acting as a hybrid of quantization and spiking.
Key results
- On LLaMA-1-7B W4A4 with the OmniQuant pipeline, SpikeLLM reports 52.48 average accuracy vs 49.93 for OmniQuant and reduces WikiText2 perplexity by 11.01%.
- On LLaMA-2-70B W2A16, SpikeLLM achieves 60.38 average accuracy vs 53.31 for OmniQuant.
- With the GPTQ pipeline on LLaMA-2-7B W4A4, SpikeLLM-GPTQ reaches 66.76 average accuracy vs 65.61 for QuaRot-GPTQ.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| saliency-based spiking (GIF + OBSpiking) on activations and KV caches, 10% salient channels at 2 steps | LLAMA-1-7B W4A4 with OmniQuant pipeline · common sense reasoning avg | avg accuracy: +2.55 | Helped |
| saliency-based spiking on weights (2-bit) at T=2 | LLAMA-2-7B W2A16 with OmniQuant pipeline · common sense reasoning avg | avg accuracy: +8.10 (41.75 to 49.85) | Helped |
| saliency-based spiking with T=4 on 5% channels | LLAMA-2-7B W2A8 · common sense reasoning avg | avg accuracy: +4.84 (36.40 to 41.24) | Helped |
| saliency-based spiking on top of QuaRot RTN | LLAMA-2-7B W4A4 · common sense reasoning avg | avg accuracy: +1.03 | Helped |
| saliency-based spiking on top of QuaRot GPTQ | LLAMA-2-7B W4A4 · common sense reasoning avg | avg accuracy: +1.15 | Helped |
| unstructured per-element spiking steps in weights instead of structured channel-wise steps (ablation) | LLAMA-2-7B W2A16 SpikeLLM T=2 · common sense reasoning avg | avg accuracy: 48.90 to 56.11 | Helped |
| adding ternary GIF weight spiking with mixed 1/2/4-step allocation vs PB-LLM binary baseline | LLAMA-2-7B additive linear layers · common sense reasoning avg | avg accuracy: 52.10 (Ter 70:25:5, 1.4 equal steps) vs 46.93 (PB-LLM 80%, 2.4 equal steps) | Helped |
| increasing ternary spiking share of high-step weights (e.g., 90:5:5 with fewer multi-step channels) (ablation) | SpikeLLM Ter on LLAMA-2-7B · common sense reasoning avg | avg accuracy: 52.10 (70:25:5) to 36.09 (90:5:5) | Hurt |
| saliency-based spiking with 2-bit weights under EfficientQAT pipeline | LLAMA-2-7B 2-bit · EfficientQAT eval avg | avg accuracy: 59.47 vs 58.65 | Helped |
Surprising findings
- Structured vs unstructured saliency spiking in weights: unstructured spiking raises LLaMA-2-7B W2A16 average from 48.90 to 56.11.
Limitations
- Clear performance gap between spiking LLMs and full-precision ANN LLMs.
- Calibration uses only 128 samples and no continual pretraining, which limits performance.
- Multi-step GIF inference costs more operations than GEMM-only quantization in the 1-bit neuromorphic workflow.
Experimental setup
Datasets: WikiText2 C4 Alpaca
Benchmarks: WikiText2 C4 PIQA ARC-easy ARC-challenge BoolQ HellaSwag Winogrande MMLU
Metrics: perplexity accuracy average accuracy ACE Sparse ACE
Baselines: OmniQuant SmoothQuant LLM-QAT QuaRot PB-LLM QLoRA IR-QLoRA QA-LoRA EfficientQAT AQLM AutoRound
Builds on: OmniQuant GPTQ Optimal Brain Surgeon SpikeLM PB-LLM
Closest papers by meaning
- SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanisms (ICML 2024 · similarity 0.92)
- SpikingBERT: Distilling BERT to Train Spiking Language Models Using Implicit Differentiation (AAAI 2024 · similarity 0.90)
- SpikingSSMs: Learning Long Sequences with Sparse and Parallel Spiking State Space Models (AAAI 2025 · similarity 0.89)
- P-SPIKESSM: HARNESSING PROBABILISTIC SPIKING STATE SPACE MODELS FOR LONG-RANGE DEPENDENCY TASKS (ICLR 2025 · similarity 0.88)
- PaceLLM: Brain-Inspired Large Language Models for Long-Context Understanding (NeurIPS 2025 · similarity 0.88)
- A Scalable, Causal, and Energy Efficient Framework for Neural Decoding with Spiking Neural Networks (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.