Research map / Architectures and efficiency / RNNs and normalization
Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes sparse BPTT for Delta RNNs that reuses forward temporal-sparsity masks in backpropagation, cutting training matrix operations by about 80% and proposing a training accelerator evaluated in RTL simulation.
Problem
Training RNNs with BPTT requires dense matrix-vector multiplications and memory accesses that are costly for online training on edge devices.
Approach
Delta RNN temporal sparsity created in the forward pass is identical in backward propagation, so gradient computation for inactivated neurons can be skipped with the same binary mask, giving mathematically identical weight updates to dense delta BPTT.
Key results
- Delta LSTM with threshold 0.1 on GSCD reaches 92.5% accuracy versus 93.1% for LSTM, with backward MACs reduced from 147.5K to 24.5K and 83.4% sparsity.
- On Fluent Speech Commands, Delta LSTM trained with sparse BPTT uses 7.3X fewer training operations than the LSTM with only a 1.155X increase in error rate.
- RTL simulation of a 16-PE accelerator shows nearly 2X, 5X, and 10X training speedup for 256I-256H Delta RNN at 50%, 80%, and 90% input sparsity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sparse instead of dense BPTT (ablation) | Delta LSTM (Θ=0.1) on GSCD · GSCD test | accuracy: 92.5 -> 92.5 | No clear effect |
| Use sparse instead of dense BPTT (ablation) | Delta LSTM (Θ=0.1) on GSCD · GSCD test | backward MACs: 147.5 -> 24.5 | Helped |
| Use sparse instead of dense BPTT (ablation) | Delta LSTM (Θ=0.2) on GSCD · GSCD test | accuracy: 91.0 -> 91.0 | No clear effect |
| Use sparse instead of dense BPTT (ablation) | Delta GRU (Θ=0.1) on GSCD · GSCD test | accuracy: 92.7 -> 92.7 | No clear effect |
| Use sparse instead of dense BPTT (ablation) | Delta GRU (Θ=0.2) on GSCD · GSCD test | accuracy: 91.6 -> 91.6 | No clear effect |
| Increase delta threshold from Θ=0.1 to Θ=0.2 (ablation) | Delta LSTM · GSCD test | accuracy: 92.5 -> 91.0 | Hurt |
| Increase delta threshold from Θ=0.1 to Θ=0.2 (ablation) | Delta LSTM · GSCD test | backward MACs: 24.5 -> 10.5 | Helped |
| Increase delta threshold from Θ=0.1 to Θ=0.2 (ablation) | Delta GRU · GSCD test | accuracy: 92.7 -> 91.6 | Hurt |
| Use Delta LSTM instead of LSTM | GSCD class-incremental, batch 32 · GSCD class-incremental, 35 | accuracy: 90.8 -> 90.5 | Hurt |
| Use Delta LSTM instead of LSTM | GSCD class-incremental, batch 1 · GSCD class-incremental, 20+3x5 | MACs: 221.2 -> 44.7 | Helped |
| Use Delta LSTM instead of LSTM | GSCD class-incremental, batch 1 · GSCD class-incremental, 20+3x5 | accuracy: 82.3 -> 80.3 | Hurt |
| Use Delta LSTM instead of LSTM | GSCD class-incremental, batch 1 · GSCD class-incremental, 20+1x15 | accuracy: 76.6 -> 74.8 | Hurt |
Surprising findings
- Unlike EGRU and spiking networks, the backward sparsity equals the forward sparsity in Delta RNNs, because the mask from the forward pass is reused directly.
Limitations
- Hardware speedup is evaluated only through logic/RTL simulation, not a fabricated chip.
- Accuracy loss grows with delta threshold; the 7.3X reduction on FSCD comes with a 1.155X error-rate increase.
- Experiments use small LSTM/GRU models (about 55k-74k parameters).
Experimental setup
Datasets: Google Speech Commands v2 Fluent Speech Commands
Benchmarks: Google Speech Commands v2 Fluent Speech Commands
Metrics: test accuracy MxV/MAC operations sparsity DRAM write accesses RTL simulation speedup
Baselines: LSTM GRU Skip RNN Zoneout EGRU
Builds on: Delta Networks BPTT iCaRL Adam
Closest papers by meaning
- Delta Networks for Optimized Recurrent Network Computation (ICML 2017 · similarity 0.91)
- Efficient recurrent architectures through activity sparsity and sparse back-propagation through time (ICLR 2023 · similarity 0.90)
- Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity (ICML 2025 · similarity 0.89)
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products (NeurIPS 2025 · similarity 0.87)
- Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN (ICLR 2024 · similarity 0.87)
- MEET: Towards Memory-Efficient Temporal Sparse Deep Neural Networks (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.