Research map / Optimization / Optimizers
Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks
Alexander Shekhovtsov, Viktor Yanush, Boris Flach
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PSA instead of ARM for low-sample gradient estimation | Stochastic binary networks · 5-5-5 Bernoulli-unit classification network | gradient estimation RMSE | Improved |
| Use PSA instead of the straight-through estimator | Stochastic binary networks · 5-5-5 Bernoulli-unit classification network | gradient estimation accuracy | Improved |
| Use PSA instead of the popular hard-tanh straight-through variant | Stochastic binary networks · CIFAR-10 | training stability | Improved |
| Use PSA instead of REINFORCE | Deep stochastic binary networks · CIFAR-10 | training progress | Improved |
| Use PSA instead of Tanh, Concrete, ADF, or LocalReparam relaxed objectives | Deep stochastic binary networks · CIFAR-10 | expected SBN training loss | Improved |
| Add the extra linearizations that derive ST from PSA (ablation) | PSA · CIFAR-10 | training performance | No clear effect |
| Add the extra linearizations that derive ST from PSA (ablation) | PSA · 5-5-5 Bernoulli-unit classification network | gradient estimation accuracy | Hurt |
| Replace the model-matched smooth derivative with a hard-tanh derivative (ablation) | Straight-through estimator for stochastic binary networks · CIFAR-10 without augmentation and with affine augmentation | training stability | Hurt |
| Increase the number of gradient-estimator samples (ablation) | Unbiased estimators · 5-5-5 Bernoulli-unit classification network | gradient estimation RMSE | Improved |
| Increase the number of gradient-estimator samples beyond the useful range (ablation) | Biased estimators · 5-5-5 Bernoulli-unit classification network | gradient estimation RMSE | Mixed |
| Increase network depth and width (ablation) | PSA and ST · Stochastic binary networks | relative advantage in gradient estimation | Improved |
Closest papers by meaning
- Low Bias Low Variance Gradient Estimates for Boolean Stochastic Networks (ICML 2020 · similarity 0.88)
- AdaSTE: An Adaptive Straight-Through Estimator To Train Binary Neural Networks (CVPR 2022 · similarity 0.87)
- Layerwise Progressive Freezing: A Training Scaffold for Depth-Scalable Binary Networks (NeurIPS 2026 · similarity 0.85)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.85)
- Training Binary Neural Networks via Gaussian Variational Inference and Low-Rank Semidefinite Programming (NeurIPS 2024 · similarity 0.84)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
