Research map / Optimization / Optimizers
Low Bias Low Variance Gradient Estimates for Boolean Stochastic Networks
Adeel Pervez, Taco Cohen, Efstratios Gavves
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Importance reweighting of Straight-Through gradient by 0.5/p(z_i) (FouST) instead of plain ST | Boolean stochastic networks (FouST) · OMNIGLOT stochastic MLPs (one and two layers) | training ELBO | Improved |
| Fourier noise operator (rho=0.5, correlated samples) applied to decoder, versus independent samples (ablation) | FouST on stochastic ResNet VAE · CIFAR-10 | training ELBO (bits/dimension) | Improved |
| Fourier noise operator applied to decoder (vs. no NoiseOp) (ablation) | FouST on stochastic ResNet VAE with 64-channel layer · CIFAR-10 | training stability | Improved |
| Decrease noise operator rho (more stable training) (ablation) | FouST with Fourier noise operator · CIFAR-10 | training stability | Improved |
| Wider 256-channel stochastic layer with noise operator, single layer (vs 4-layer FouST) | FouST · CIFAR-10 | validation bits per dimension: 4.16 -> 4.02 bpd | Improved |
| Replace unbiased REBAR with FouST (biased, single decoder evaluation) | two-layer stochastic MLP on MNIST · MNIST test ELBO | test ELBO: -94.43 vs -91.94 | Improved |
| Uniform-sampling/discounting Taylor terms and representation rescaling tau for FouST | Boolean stochastic networks · toy problem minimizing E[(z-t)^2] | convergence to optimal p(z=+1)=0 | Improved |
Closest papers by meaning
- Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks (NeurIPS 2020 · similarity 0.88)
- On the Overlooked Structure of Stochastic Gradients (NeurIPS 2023 · similarity 0.87)
- Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity (NeurIPS 2021 · similarity 0.87)
- Gradient Estimation with Stochastic Softmax Tricks (NeurIPS 2020 · similarity 0.86)
- On the Bias-Variance-Cost Tradeoff of Stochastic Optimization (NeurIPS 2021 · similarity 0.86)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
