Research map / Architectures and efficiency / Pruning and efficient tuning
HADAMRNN: BINARY AND SPARSE TERNARY ORTHOGONAL RNNS
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HadamRNN and Block-HadamRNN, orthogonal RNNs whose recurrent weights are binary or sparse ternary, built by sign-flipping rows of Sylvester Hadamard matrices and trained with the straight-through estimator.
Problem
Vanilla and orthogonal RNNs are highly sensitive to recurrent weight changes, so binarizing or ternarizing recurrent weights has not succeeded; prior quantized ORNNs need at least 4-5 bits and quantized LSTMs fail long-horizon tasks such as the 1000-step copy task.
Approach
Parameterize binary orthogonal recurrent matrices as diag(u) times a normalized Sylvester Hadamard matrix with trainable binary sign vector u (and sparse ternary block-diagonal variants with I_q kron Hadamard), so orthogonality holds by construction and u is trained with STE.
Key results
- HadamRNN with 1-bit recurrent weights solves the 1000-step copy task (cross-entropy 1.6e-7, 1.40 kB), beating 5-bit QORNN (2.5e-3, 50.6 kB).
- On pMNIST, fully quantized HadamRNN reaches 94.90% at 3.58 kB, versus 94.76% for 8-bit QORNN at 35.0 kB and 97.00% for full-precision ORNN at 1046 kB.
- On IMDB, HadamRNN (d_h=512, 1-bit W, 4-bit U/V) reaches 87.43% at 130.32 kB, vs 84.02% for full-precision ORNN (d_h=128).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use binary orthogonal recurrent weights instead of 5-bit QORNN weights | HadamRNN versus QORNN · Copy task (T=1000) | cross-entropy: 2.5e-3 -> 1.6e-7 | Helped |
| Quantize activations to 12 bits (ablation) | HadamRNN · Permuted MNIST | accuracy: 94.88 -> 94.90 | No clear effect |
| Quantize activations to 12 bits (ablation) | HadamRNN · Sequential MNIST | accuracy: 96.63 -> 96.34 | Hurt |
| Increase U and V bitwidth from 2 to 4 bits (ablation) | HadamRNN · Permuted MNIST | accuracy: 91.13 -> 94.88 | Helped |
| Increase U and V bitwidth from 2 to 4 bits (ablation) | HadamRNN · Sequential MNIST | accuracy: 92.65 -> 96.63 | Helped |
| Increase hidden size from 128 to 512 (ablation) | HadamRNN · IMDB | accuracy: 85.34 -> 87.43 | Helped |
| Use ternary rather than 2-bit U and V (ablation) | HadamRNN (hidden size 128) · IMDB | accuracy: 85.34 -> 81.18 | Hurt |
| Use sparse ternary recurrent weights with q=8 instead of binary weights (ablation) | Block-HadamRNN versus HadamRNN · Copy task (T=1000) | cross-entropy: 1.6e-7 -> 6.6e-5 | Hurt |
| Use sparse ternary recurrent weights with q=2 instead of binary weights (ablation) | Block-HadamRNN versus HadamRNN · Permuted MNIST | accuracy: 94.88 -> 93.12 | Hurt |
| Use sparse ternary recurrent weights with q=2 instead of binary weights (ablation) | Block-HadamRNN versus HadamRNN · Sequential MNIST | accuracy: 96.63 -> 96.47 | Hurt |
| Use sparse ternary recurrent weights with q=2 instead of binary weights (ablation) | Block-HadamRNN versus HadamRNN · IMDB | accuracy: 87.43 -> 86.30 | Hurt |
| Increase q from 2 to 128 (ablation) | Block-HadamRNN · Permuted MNIST | accuracy: 93.12 -> 60.74 | Hurt |
Surprising findings
- Block-HadamRNN with q=8 solves the copy task with far fewer additions (2048 vs 16384) than HadamRNN.
- Binary-weight LSTMs with 2 bits at d_h=512 only slightly outperform HadamRNN on IMDB despite being much larger.
Limitations
- On sMNIST and pMNIST with 2-bit U and V, accuracy drops substantially (e.g., 91.13% on pMNIST), and binary-recurrent models underperform LSTMs on sMNIST.
- Results on GLUE are limited to training from scratch on single tasks, without pretrained embeddings, and lag behind pretrained BERT-based models.
- Quantization of activations is only applied post-training, not during training.
Experimental setup
Datasets: copy task (synthetic) permuted MNIST sequential MNIST IMDB SST-2 QQP
Benchmarks: copy task permuted MNIST sequential MNIST IMDB SST-2 QQP HAR-2 DSA-19
Metrics: accuracy cross-entropy model size (kB) additions count
Baselines: LSTM ORNN QORNN FastGRNN BiBERT BinaryBERT TernaryBERT
Builds on: orthogonal RNN QORNN straight-through estimator BinaryConnect
Closest papers by meaning
- An Efficient Matrix Multiplication Algorithm for Accelerating Inference in Binary and Ternary Neural Networks (ICML 2025 · similarity 0.86)
- Learning Recurrent Binary/Ternary Weights (ICLR 2019 · similarity 0.86)
- Orthogonal Recurrent Neural Networks with Scaled Cayley Transform (ICML 2018 · similarity 0.86)
- HitNet: Hybrid Ternary Recurrent Neural Network (NeurIPS 2018 · similarity 0.86)
- Training Binary Neural Networks via Gaussian Variational Inference and Low-Rank Semidefinite Programming (NeurIPS 2024 · similarity 0.86)
- QBB: Quantization with Binary Bases for LLMs (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.