Research map / Architectures and efficiency / Pruning and efficient tuning
Exploring Sparsity in Recurrent Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| gradually prune an RNN with 1760 hidden units during training | Deep Speech 2 · English speech devset | CER: 10.67 -> 12.88 | Hurt |
| increase model width to 2560 units and gradually prune | Deep Speech 2 · English speech devset | CER: 10.67 -> 10.59 | Helped |
| increase model width to 3072 units and gradually prune | Deep Speech 2 · English speech devset | CER: 10.67 -> 10.25 | Helped |
| use a dense model with 704 hidden units instead of a gradually pruned model with 1760 units (ablation) | Deep Speech 2 · English speech devset | CER: 12.88 -> 14.50 | Hurt |
| replace gradual pruning with hard threshold pruning (ablation) | RNN Dense Baseline · English speech devset | CER: 7% to 9% better for gradual pruning at approximately the same number of parameters | Hurt |
| gradually prune a GRU with 2560 hidden units | GRU · English speech devset | CER: 9.55 -> 10.87 | Hurt |
| increase GRU width to 3568 units and gradually prune | GRU · English speech devset | CER: 9.55 -> 9.76 | Hurt |
| prune only GRU layers and leave fully connected layers dense (ablation) | GRU · English speech devset | CER: around 7% worse than the baseline dense model | Hurt |
| use 95% sparsity for a 1760-unit RNN layer | RNN GEMM · RNN layer, 1760 units | speedup: 1 -> 2.8 | Helped |
| use 95% sparsity for a 2560-unit GRU layer | GRU GEMM · GRU layer, 2560 units | speedup: 1 -> 6.80 | Helped |
| make sparsity equal across recurrent layers rather than pruning final layers less (ablation) | Deep Speech 2 · English speech devset | CER | Hurt |
| use a convex pruning schedule instead of a linear-slope schedule (ablation) | RNN pruning · English speech devset | CER | Helped |
Closest papers by meaning
- One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation (ICLR 2020 · similarity 0.92)
- Computation and Memory-Efficient Model Compression with Gradient Reweighting (NeurIPS 2025 · similarity 0.90)
- Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training (AAAI 2024 · similarity 0.88)
- Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity (ICML 2025 · similarity 0.88)
- Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN (ICLR 2024 · similarity 0.88)
- Targeted Low-rank Refinement: Enhancing Sparse Language Models with Precision (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.