Research map / Architectures and efficiency / RNNs and normalization
Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Implements sparse BPTT training of spiking neural networks on the Graphcore IPU's MIMD distributed-memory architecture, reporting 5-10x (up to ~38x at high sparsity) throughput gains over NVIDIA A100 GPUs without convergence loss.
Problem
SNNs and other recurrent, sparse models cannot exploit SIMD GPU/TPU hardware efficiently, making their training slow and expensive compared to dense Transformers and CNNs.
Approach
Distribute LIF neurons, weights and gradients tile-locally on a MIMD IPU and pass only binary sparse spike index tensors between tiles, with a custom sparse matrix-vector read-and-sum forward and sparse backward pass for BPTT training.
Key results
- Sparse IPU training achieves at least 5-10x throughput over A100 GPU dense implementation, with 15-20x at more extreme sparsity levels.
- NMNIST throughput of 14.72 thousand sequences per second on IPU vs 1.02 thousand on A100 at ~97.8% communication sparsity.
- Sparse training shows no reduction in final test accuracy versus dense implementations on NMNIST, DVSGesture and SHD; scaling to 16 IPUs gives a 1.4x slowdown.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sparse activations and sparse matrix multiplication on the IPU instead of a dense GPU implementation | SNN training · Training workloads | throughput: 5-10× | Helped |
| Use sparse activations and sparse matrix multiplication on the IPU instead of a dense GPU implementation | SNN training · Higher activation sparsity | throughput: 15-20× | Helped |
| Use sparse activation tensors instead of dense activation tensors | SNN training · NMNIST, DVSGesture, and SHD | test accuracy | No clear effect |
| Use sparse activation tensors instead of dense activation tensors | SNN training · NMNIST, DVSGesture, and SHD | training convergence | No clear effect |
| Increase network size per IPU | Sparse SNN on IPU · Single-IPU scaling | acceleration vs. GPU: 30-40× | Helped |
| Increase computational workload relative to communication latency by increasing batch size | Sparse SNN across IPUs · Multi-IPU weak scaling | throughput slowdown: 1.2× | Helped |
| Increase network density per IPU | Sparse SNN across IPUs · 16-IPU weak scaling | throughput slowdown: 1.9× to 1.5× | Helped |
Surprising findings
- Sparse activation training sometimes yields the best test accuracy across all three datasets, suggesting an implicit dropout-like regularization effect.
- Prior IPU SNN studies reported non-favorable IPU performance, which this work reverses through tile-local and sparse implementation.
Limitations
- Memory constraints of BPTT limit scaling to larger networks on IPU's limited SRAM.
- Sparse spike tensors have a fixed maximal size N_max and excess spikes are randomly dropped, making the algorithm inexact.
- Convergence experiments were run in Jax with custom CUDA kernels rather than on the IPU, and some scaling experiments used reduced timesteps and SGD.
Experimental setup
Datasets: SHD N-MNIST DVSGesture
Benchmarks: SHD N-MNIST DVSGesture
Metrics: throughput acceleration factor test accuracy training loss convergence
Baselines: dense GPU implementation on NVIDIA A100 NVIDIA V100 NVIDIA RTX 3090
Builds on: surrogate gradient SuperSpike backpropagation through time Adam
Closest papers by meaning
- Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity (ICML 2025 · similarity 0.89)
- Improving the Sparse Structure Learning of Spiking Neural Networks from the View of Compression Efficiency (ICLR 2025 · similarity 0.87)
- DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs (NeurIPS 2025 · similarity 0.86)
- FlashRNN: I/O-Aware Optimization of Traditional RNNs on modern hardware (ICLR 2025 · similarity 0.86)
- Parallelizing non-linear sequential models over the sequence length (ICLR 2024 · similarity 0.86)
- Accelerating neural network training: An analysis of the AlgoPerf competition (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.