Research map / Architectures and efficiency / RNNs and normalization
Understanding weight-magnitude hyperparameters in training binary networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace latent-weight SGD with the equivalent filtering-based optimizer | Bi-RealNet-20 · CIFAR-10 | Top-1 Acc(%): 85.0 -> 86.5; 86.7 | No clear effect |
| Use the filtering-based optimizer instead of two-step training | ReActNet-A · ImageNet | Top-1 Acc(%): 70.5 -> 69.7 | Hurt |
| Use the filtering-based optimizer instead of two-step training | ReActNet-A · ImageNet | Top-5 Acc(%): 89.1 -> 88.9 | Hurt |
| Use zero latent-weight initialization with stochastic signs at zero (ablation) | Magnitude-independent BNN optimization · CIFAR-10 | accuracy | Helped |
| Replace a first-order gradient filter with a second-order filter (ablation) | Binary weight optimization · CIFAR-10 | gradient noise | Helped |
| Remove alpha decay (ablation) | Filtering-based BNN optimizer · CIFAR-10 | accuracy | Hurt |
| Vary alpha (ablation) | Filtering-based BNN optimizer · CIFAR-10 | accuracy | Mixed |
| Scale the learning rate and inversely scale initialization (ablation) | Magnitude-independent BNN optimization · CIFAR-10 | bit flipping ratio | No clear effect |
| Scale both learning rate and initialization without changing their ratio (ablation) | Magnitude-independent BNN optimization · CIFAR-10 | training | No clear effect |
| Use a smaller learning rate with fixed initial weights (ablation) | Magnitude-independent BNN optimization · CIFAR-10 | accuracy | Hurt |
| Use standard magnitude-dependent clipping and scaling rather than magnitude-independent optimization (ablation) | BNN optimization · CIFAR-10 | accuracy | Mixed |
| Omit learning-rate decay in the audio hyperparameter search (ablation) | Latent-weight and filtering perspectives · Audio classification | accuracy | No clear effect |
Closest papers by meaning
- Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization (NeurIPS 2019 · similarity 0.88)
- The High-Dimensional Geometry of Binary Neural Networks (ICLR 2018 · similarity 0.87)
- Resilient Binary Neural Network (AAAI 2023 · similarity 0.86)
- Fast and Slow Gradient Approximation for Binary Neural Network Optimization (AAAI 2025 · similarity 0.86)
- Training Binary Neural Networks via Gaussian Variational Inference and Low-Rank Semidefinite Programming (NeurIPS 2024 · similarity 0.85)
- Magnitude Invariant Parametrizations Improve Hypernetwork Learning (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.