Research map / Architectures and efficiency / RNNs and normalization
Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compresses S5 linear RNNs via iterative magnitude pruning to 90% weight sparsity, ReLU-based activation sparsification, and W8A16 QAT, then deploys on Intel Loihi 2 for streaming audio denoising.
Problem
Deploying linear RNNs for streaming edge applications requires hardware-aware compression; the efficiency-performance trade-off of extremely sparse (>=90%) linear RNNs versus dense models under equal inference compute budgets is under-explored.
Approach
Combine iterative magnitude pruning of weights to very high sparsity with ReLU-fication to sparsify activations in the S5 linear RNN, plus quantization-aware training to fixed-point, and map the resulting model onto an event-driven neuromorphic chip to convert sparsity into latency and energy gains.
Key results
- Sparse-8 S5 model requires 2x less compute and 36% lower memory than dense-3 at the same accuracy.
- Sparse-11 S5 achieves 3.2x less compute and 5.37x lower memory iso-accuracy versus Spiking-FullSubNet-XL (15.2 dB SI-SNR).
- On Loihi 2 with token-by-token processing, latency is 42x lower and energy 149x lower per token than a dense model on Jetson Orin Nano.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add weight pruning and ReLU-based activation sparsity, scaling the model to Sparse-8 | S5 · Intel N-DNS test | effective compute: 2× less compute | Helped |
| Add weight pruning and ReLU-based activation sparsity, scaling the model to Sparse-8 | S5 · Intel N-DNS test | memory footprint: 36% lower memory | Helped |
| Add weight pruning and ReLU-based activation sparsity, scaling the model to Sparse-11 | S5 · Intel N-DNS test | effective compute: 3.2× less compute | Helped |
| Add weight pruning and ReLU-based activation sparsity, scaling the model to Sparse-11 | S5 · Intel N-DNS test | memory footprint: 5.37× lower memory | Helped |
| Deploy sparse S5 with W8A16 fixed-point arithmetic on Loihi 2 instead of dense S5 on Jetson Orin Nano | Sparse-8 S5 versus Dense-3 S5 · Intel N-DNS test, token-by-token with real-time chunks | latency: 42× faster | Helped |
| Deploy sparse S5 with W8A16 fixed-point arithmetic on Loihi 2 instead of dense S5 on Jetson Orin Nano | Sparse-8 S5 versus Dense-3 S5 · Intel N-DNS test, token-by-token with real-time chunks | energy: 149 × less energy | Helped |
| Increase batch size to b=b_{\text{max}} (ablation) | Jetson Orin Nano Dense-3 S5 · Intel N-DNS test, sample-by-sample | throughput: \qty{4.28}{\sample/\second} -> \qty{1130.09}{\sample/\second} | Helped |
| Apply sparsification to an under-parameterized dense-3 model (ablation) | S5 · Intel N-DNS test | SI-SNR: 7.3% | Hurt |
| Scale model dimensions to use a larger sparse model instead of dense-3 | S5 · Intel N-DNS test | SI-SNR | No clear effect |
| Add synaptic sparsity to a ReLU-fied model (ablation) | S5 model variant 6 · Intel N-DNS test | activation sparsity | Hurt |
| Replace GELU with ReLU and add ReLUs after residual addition and to the real S5 hidden state (ablation) | S5 · Intel N-DNS test | effective MACs | Helped |
| Use QAT rather than PTQ (ablation) | Sparse-6 S5 · Intel N-DNS test | SI-SNR after static quantization | Helped |
Surprising findings
- Sparsifying under-parameterized dense models substantially degrades performance (dense-3 sparsified loses 7.3% SI-SNR), but larger sparse models recover accuracy.
- Synaptic sparsity lowers activation sparsity across layers, indicating the model compensates for pruning with higher activity.
Limitations
- Jetson baseline runs FP32 while Loihi 2 runs W8A16, so comparison is across different precisions; fixed-point JAX model gave no speedup on Jetson.
- Fixed-point and Loihi 2 outputs show SI-SNR degradation from simulation mismatch, with error accumulating with depth.
- Scaling to language or multimodal tasks is untested; only audio denoising and keyword spotting evaluated.
Experimental setup
Datasets: Intel N-DNS Challenge (derived from Microsoft DNS Challenge) SpeechCommands V2-35
Benchmarks: Intel N-DNS Challenge test set SpeechCommands V2-35
Metrics: SI-SNR test accuracy effective MACs memory footprint latency energy per token throughput
Baselines: Spiking-FullSubNet-XL dense S5 models Jetson Orin Nano (FP32 dense S5)
Builds on: S5 iterative magnitude pruning ReLU-fication quantization-aware training JaxPruner
Closest papers by meaning
- Sparse Persistent RNNs: Squeezing Large Recurrent Networks On-Chip (ICLR 2018 · similarity 0.91)
- Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training (AAAI 2024 · similarity 0.89)
- Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models (AAAI 2024 · similarity 0.89)
- LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Units (ICLR 2024 · similarity 0.88)
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.87)
- Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.