Research map / Architectures and efficiency / RNNs and normalization
meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply meProp (top-k sparsified backprop, updating 1-4% of weights) | LSTM / MLP across POS-Tag, Parsing, MNIST · POS-Tag, Parsing, MNIST (AdaGrad and Adam) | test accuracy/UAS: +0.03 to +0.48 in most settings; -0.04 and -0.10 dev-only drops | Helped |
| apply meProp top-k backprop | LSTM POS-Tag with Adam · POS-Tag (Adam) | dev accuracy: -0.04 | Hurt |
| apply meProp top-k backprop | LSTM POS-Tag · POS-Tag (AdaGrad/Adam) | backprop time speedup: 69.2x and 65.4x | Helped |
| apply meProp top-k backprop | MLP parsing and MNIST · Parsing and MNIST (CPU) | backprop time speedup: 18.1x to 41.7x | Helped |
| top-k selection of gradients vs random selection of same number of elements (ablation) | MLP meProp · MNIST | test accuracy: top-k consistently better than random | Helped |
| reduce hidden dimension to k (no sparsification) instead of meProp (ablation) | LSTM and MLP · POS-Tag, Parsing, MNIST (Adam) | test accuracy/UAS: MNIST 95.77 vs 98.01; Parsing 88.37 vs 90.01 | Hurt |
| add meProp on top of dropout (ablation) | LSTM, MLP with dropout · POS-Tag, Parsing, MNIST (Adam) | test accuracy/UAS: +0.11 POS, +0.46 UAS parsing, +0.23 MNIST | Helped |
| apply meProp with 2-5 hidden layers (ablation) | MLP · MNIST (Adam) | test accuracy: +0.05 to +0.16 across 2-5 layers | Helped |
| apply meProp unified top-k on whole mini-batch (GPU) (ablation) | MLP (h=8192) on GPU · MNIST GPU overall backprop | overall backprop time: 11.8x at k=8; 3.6x at k=512 | Helped |
| apply meProp on GPU speedup with small hidden dimension | MLP h=64/512 on GPU · GPU speed | speedup: speedup less significant at low hidden dimension | No clear effect |
Closest papers by meaning
- SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks at the Edge (ICML 2023 · similarity 0.91)
- Efficient Neural Network Training via Forward and Backward Propagation Sparsification (NeurIPS 2021 · similarity 0.90)
- ReSprop: Reuse Sparsified Backpropagation (CVPR 2020 · similarity 0.88)
- Accelerated training through iterative gradient propagation along the residual path (ICLR 2025 · similarity 0.87)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.87)
- Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks using the Marginal Likelihood (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.