Research map / Architectures and efficiency / RNNs and normalization
Learning Recurrent Binary/Ternary Weights
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add batch normalization to the recurrent vector-matrix products while learning binary or ternary weights | LSTM recurrent weight quantization · Sequential tasks | prediction performance | Helped |
| Use ternary rather than binary recurrent weights | LSTM · Text8 | BPC: 1.54 -> 1.51 | Helped |
| Use ternary rather than binary recurrent weights | LSTM · Penn Treebank word-level, large | Perplexity: 76.5 -> 76.3 | Helped |
| Use binary recurrent weights | LSTM · Penn Treebank word-level, medium | Perplexity: 87.6 -> 87.2 | Helped |
| Use ternary recurrent weights | LSTM · MNIST pixel-by-pixel | accuracy: 98.9 -> 98.8 | No clear effect |
| Use ternary recurrent weights | Attentive Reader · CNN question answering | accuracy: 59.81 -> 60.03 | Helped |
| Replace full-precision weights with binary recurrent weights | Low-Power ASIC · Recurrent inference | Power (mW): 336 -> 37 | Helped |
| Replace full-precision weights with binary recurrent weights | High-Speed ASIC · Recurrent inference | Throughput (GOps/sec): 80 -> 800 | Helped |
| Use smaller batch sizes for batch-normalized weight quantization | LSTM · Penn Treebank character-level modeling | prediction accuracy | Hurt |
Closest papers by meaning
- Alternating Multi-bit Quantization for Recurrent Neural Networks (ICLR 2018 · similarity 0.89)
- Parallelizing Linear Recurrent Neural Nets Over Sequence Length (ICLR 2018 · similarity 0.89)
- Fast-Slow Recurrent Neural Networks (NeurIPS 2017 · similarity 0.88)
- Quasi-Recurrent Neural Networks (ICLR 2017 · similarity 0.88)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.88)
- Learning Useful Representations of Recurrent Neural Network Weight Matrices (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.