Research map / Architectures and efficiency / RNNs and normalization
Delta Networks for Optimized Recurrent Network Computation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Run the standard GRU as a delta network | TIDIGITS GRU · TIDIGITS | accuracy: 96% -> 95% | Hurt |
| Add activation rounding during training (ablation) | TIDIGITS delta network · TIDIGITS | accuracy: 95% -> nearly 97% | Helped |
| Add Gaussian noise to activations during training (ablation) | TIDIGITS delta network · TIDIGITS | accuracy: nearly 97% -> about 97% | Helped |
| Train directly using the delta network model (ablation) | TIDIGITS GRU · TIDIGITS | speedup: 4.2X -> 8X | Helped |
| Train directly using the delta network model (ablation) | TIDIGITS GRU · TIDIGITS | accuracy: about 97% -> 98% | Helped |
| Account for pre-existing weight sparsity (ablation) | TIDIGITS trained delta network · TIDIGITS | speedup: 8X -> 10X | Helped |
| Account for pre-existing weight sparsity (ablation) | TIDIGITS trained delta network · TIDIGITS | accuracy | No clear effect |
| Add an L1 cost on network changes (ablation) | TIDIGITS trained delta network · TIDIGITS | speedup: 10X -> 11.9X | Helped |
| Add an L1 cost on network changes (ablation) | TIDIGITS trained delta network · TIDIGITS | accuracy: 98% -> 97% | Hurt |
| Train the RNN to run as a delta network (ablation) | Wall Street Journal RNN · Wall Street Journal | speedup: 5.5 -> 5.7X | Helped |
| Run an existing RNN as a delta network at Θ=0.2 | Wall Street Journal RNN · Wall Street Journal | WER: 10.2% -> 10.8% | Hurt |
| Use a delta network GRU on temporally redundant CNN features | Comma.ai CNN-RNN · Comma.ai driving validation | RNN compute speedup: about 100X | Helped |
Closest papers by meaning
- Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training (AAAI 2024 · similarity 0.91)
- DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products (NeurIPS 2025 · similarity 0.88)
- Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR 2025 · similarity 0.86)
- Improving Bilinear RNN with Closed-loop Control (NeurIPS 2025 · similarity 0.85)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.85)
- MEET: Towards Memory-Efficient Temporal Sparse Deep Neural Networks (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.