AI Paper Atlas

Research map / Optimization / Optimizers

Enhancing Optimization Robustness in 1-bit Neural Networks through Stochastic Sign Descent

NianHui Guo, Hong Guo, Christoph Meinel, Haojin Yang

ECCV 2024 · Optimization › Optimizers · approach: Adam convergence and adaptive learning rates · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace Adam with the Diode sign-descent optimizer on BNext-18†BNext-18† · ImageNet ILSVRC2012 valTop-1 accuracy (%): 67.5 -> 68.2Improved
replace Adam with Diode on ReActNet-A without two-stage trainingReActNet-A · ImageNet ILSVRC2012 valTop-1 accuracy (%): 69.4 -> 69.6Improved
replace Adam with Diode on binary BERT (BiT‡) without data augmentationBiT · GLUEaverage score: 71.0 -> 74.7Improved
use BOP or Gradient Filter latent-weight-free optimizers instead of Adam on binary BERT (ablation)BiT · GLUEaverage score: 71.0 -> 61.9 (BOP), 62.5 (Gradient Filter)Hurt
initialize Diode from pre-trained BiT weights (ablation)BiT · GLUEaverage score: 74.7 -> 75.7Improved
add data augmentation for Diode on BiT† (ablation)BiT · GLUEaverage score: 75.7 -> 78.8Improved
increase long-term EMA ratio beta from 0.999 to 0.9999 (ablation)Diode · CIFAR100 BNext-18†Top-1 accuracy (%): 72.01 -> 74.51 (alpha=0.99)Improved
vary initial learning rate eta from 1.0 to 1e-4 (ablation)Diode · CIFAR100 BNext-18†Top-1 accuracy (%): 74.50 vs 74.45No clear effect
use CosineLR eta decay instead of step or exponential decayDiode · CIFAR100 BNext-18†validation errorImproved
train with batch size 4096 using Diode instead of best Adam (B=512) (ablation)BNext-18† · ImageNet ILSVRC2012 valTop-1 accuracy (%): +0.96% with 8x fewer iterationsImproved
use Diode with small batch size (below 128)BNext-18† · ImageNet ILSVRC2012 valTop-1 accuracy (%)Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy