AI Paper Atlas

Research map / Optimization / Optimizers

Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks

Alexander Shekhovtsov, Viktor Yanush, Boris Flach

NeurIPS 2020 · Optimization › Optimizers · approach: Stochastic gradient MCMC · 3 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use PSA instead of ARM for low-sample gradient estimationStochastic binary networks · 5-5-5 Bernoulli-unit classification networkgradient estimation RMSEImproved
Use PSA instead of the straight-through estimatorStochastic binary networks · 5-5-5 Bernoulli-unit classification networkgradient estimation accuracyImproved
Use PSA instead of the popular hard-tanh straight-through variantStochastic binary networks · CIFAR-10training stabilityImproved
Use PSA instead of REINFORCEDeep stochastic binary networks · CIFAR-10training progressImproved
Use PSA instead of Tanh, Concrete, ADF, or LocalReparam relaxed objectivesDeep stochastic binary networks · CIFAR-10expected SBN training lossImproved
Add the extra linearizations that derive ST from PSA (ablation)PSA · CIFAR-10training performanceNo clear effect
Add the extra linearizations that derive ST from PSA (ablation)PSA · 5-5-5 Bernoulli-unit classification networkgradient estimation accuracyHurt
Replace the model-matched smooth derivative with a hard-tanh derivative (ablation)Straight-through estimator for stochastic binary networks · CIFAR-10 without augmentation and with affine augmentationtraining stabilityHurt
Increase the number of gradient-estimator samples (ablation)Unbiased estimators · 5-5-5 Bernoulli-unit classification networkgradient estimation RMSEImproved
Increase the number of gradient-estimator samples beyond the useful range (ablation)Biased estimators · 5-5-5 Bernoulli-unit classification networkgradient estimation RMSEMixed
Increase network depth and width (ablation)PSA and ST · Stochastic binary networksrelative advantage in gradient estimationImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy