Research map / Optimization / Optimizers
Fast and Slow Gradient Approximation for Binary Neural Network Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Fast and Slow Gradient Generation (FSG), a hypernetwork-based gradient estimator for binary neural networks that combines a Mamba-based slow net over stored historical gradients (with layer embeddings) and an MLP fast net over current gradients.
Problem
Gradient mismatch from the non-differentiable quantization (sign) function in BNN training; existing hypernetwork gradient estimators use only current gradients and ignore historical gradient information, accumulating estimation errors.
Approach
Adds a Historical Gradient Storage module that feeds recent gradient sequences into a Mamba-based slow-net (with learnable layer recognition embeddings) to generate a momentum-like gradient, combined with an MLP fast-net output over the current gradient, to form the final gradient replacing the STE derivative.
Key results
- On CIFAR-100 with ResNet-56 and Adam, FSG reaches 69.48% test accuracy versus 68.94% for IR-Net (second best) and 64.78% for STE.
- On CIFAR-10 with ResNet-44 and Adam, FSG achieves 92.78% test accuracy versus 91.91% for LSTMFC.
- Replacing LSTM with Mamba as slow-net improves ResNet-44 CIFAR-10 test accuracy from 92.07% to 92.63%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace IR-Net with FSG | ResNet-20 · CIFAR-10 | test accuracy (%): 90.80 ± 0.971 -> 91.00 ± 0.804 | Helped |
| replace IR-Net with FSG | ResNet-44 · CIFAR-10 | test accuracy (%): 92.37 ± 0.532 -> 92.78 ± 0.053 | Helped |
| replace IR-Net with FSG | ResNet-56 · CIFAR-100 | test accuracy (%): 68.94 ± 0.967 -> 69.48 ± 0.979 | Helped |
| use Mamba instead of LSTM as the slow-net (ablation) | FSG · CIFAR-10, ResNet-44 | test accuracy (%): 92.07 ± 0.066 -> 92.63 ± 0.052 | Helped |
| use Mamba instead of LSTM as the slow-net (ablation) | FSG · CIFAR-100, ResNet-56 | test accuracy (%): 67.60 ± 0.670 -> 68.04 ± 0.948 | Helped |
| set the combination parameter β to 0.3 (ablation) | FSG · CIFAR-100, ResNet-56 | test accuracy (%) | Helped |
| set historical gradient memory length l to 6 (ablation) | FSG · CIFAR-100, ResNet-56 | test accuracy (%) | Helped |
| add the slow-net to STE | STE · CIFAR-100, ResNet-56 | seconds/epoch: 42.28 -> 196.47 | Hurt |
Surprising findings
- Adding the slow-net increases training time about 3.2x over STE per the overhead table, while inference cost is unchanged since hypernetworks are removed at deployment.
Limitations
- Evaluated only on ResNet models on CIFAR-10/100; not extended to Transformers or LLMs.
- Training cost is substantially higher than STE due to hypernetworks.
- Convergence theorem relies on strong assumptions (convexity, bounded iterates and gradients) that do not match the non-convex BNN setting.
Experimental setup
Datasets: CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 CIFAR-100
Metrics: test accuracy train accuracy cross-entropy loss seconds per epoch
Baselines: STE FCGrad LSTMFC DoReFa ReSTE RBNN IR-Net
Builds on: STE hypernetwork based gradient estimation (FCGrad) LSTMFC Mamba SGD with momentum
Closest papers by meaning
- Training Binary Neural Networks via Gaussian Variational Inference and Low-Rank Semidefinite Programming (NeurIPS 2024 · similarity 0.87)
- Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers (NeurIPS 2023 · similarity 0.86)
- How Do Adam and Training Strategies Help BNNs Optimization (ICML 2021 · similarity 0.86)
- Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimization (NeurIPS 2024 · similarity 0.85)
- Magnitude Invariant Parametrizations Improve Hypernetwork Learning (ICLR 2024 · similarity 0.85)
- BiPer: Binary Neural Networks using a Periodic Function (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.