Research map / Optimization / Optimizers
SURGE: Surrogate Gradient Adaptation in Binary Neural Networks
Haoyu Huang, Boyu Liu, Linlin Yang, Yanjing Li, Yuguang Yang, Xuhui Liu, Canyu Chen, Zhongqian Fu, Baochang Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Dual-Path Gradient Compensator (ablation) | binarized ResNet-20 baseline · CIFAR-10 | accuracy: 87.4 -> 87.8 | Improved |
| add Adaptive Gradient Scaler on top of DPGC (ablation) | SURGE ResNet-20 · CIFAR-10 | accuracy: 87.8 -> 88.0 | Improved |
| use fixed scaling factor instead of adaptive scaling (ablation) | SURGE ResNet-20 · CIFAR-10 | accuracy: fixed 0.07: 70.32% vs adaptive 87.95% | Hurt |
| compensate only gradients outside the clipping range (ablation) | SURGE ResNet-20 · CIFAR-10 | accuracy: 87.4 -> 87.7 | Improved |
| compensate only within-range gradients (ablation) | SURGE ResNet-20 · CIFAR-10 | accuracy: 87.4 -> 87.6 | Improved |
| compensate all gradients (clipped and unclipped) (ablation) | SURGE ResNet-20 · CIFAR-10 | accuracy: 87.4 -> 88.0 | Improved |
| add explicit alignment loss between auxiliary and binary branch outputs (ablation) | BiT+SURGE · GLUE dev | average score: 72.0 -> 71.2 | Hurt |
| add SURGE to BiT under same wall-clock budget | BiT · GLUE dev (fixed wall-clock budget) | average score: 65.2 -> 66.7 | Improved |
| add SURGE to BiT | BiT BERT-base · GLUE dev | average score: 70.6 -> 72.0 | Improved |
| add SURGE vs best prior on one-stage ImageNet | binarized ResNet-18 · ImageNet-1K (one-stage training) | top-1 accuracy: 61.4 -> 62.0 | Improved |
| add SURGE vs best prior on two-stage ImageNet | binarized ResNet-18 · ImageNet-1K (two-stage training) | top-1 accuracy: 66.5 -> 66.7 | Improved |
| use 1x1 instead of 3x3 auxiliary convolutions (SURGE*) (ablation) | SURGE ResNet-18 · ImageNet-1K (10 epochs) | accuracy: 47.28 -> 47.21 | No clear effect |
Closest papers by meaning
- AdaSTE: An Adaptive Straight-Through Estimator To Train Binary Neural Networks (CVPR 2022 · similarity 0.87)
- Gradient Matters: Designing Binarized Neural Networks via Enhanced Information-Flow (TPAMI 2021 · similarity 0.87)
- An Empirical study of Binary Neural Networks' Optimisation (ICLR 2019 · similarity 0.86)
- Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Training (ICCV 2023 · similarity 0.86)
- Fast and Slow Gradient Approximation for Binary Neural Network Optimization (AAAI 2025 · similarity 0.86)
- Training Binary Neural Networks via Gaussian Variational Inference and Low-Rank Semidefinite Programming (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
