Research map / Probabilistic and scientific modeling / Variational and generative sampling
Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace REBAR's fixed relaxation surrogate with a learned neural control variate | Discrete VAE training · MNIST and Omniglot linear one- and two-layer VAEs | training ELBO: MNIST: -111.6 -> -111.20; -98.22 -> -98.00. Omniglot: -116.63 -> -116.57; -108.71 -> -108.54 | Helped |
| Replace REBAR's fixed relaxation surrogate with a learned neural control variate | Discrete VAE training · MNIST and Omniglot nonlinear VAEs | validation ELBO: MNIST: -111.12 -> 119.19; Omniglot: -127.51 -> 128.20 | Hurt |
| Use RELAX instead of REBAR | Linear discrete VAEs · MNIST and Omniglot linear VAEs | epochs to REBAR best validation score: MNIST: 857 -> 531; 900 -> 620. Omniglot: 2086 -> 566; 1027 -> 673 | Helped |
| Use an action-conditional LAX/RELAX estimator instead of A2C | Reinforcement-learning policies · Cart Pole | episodes to solve: 1152 ± 90 -> 472 ± 114 | Helped |
| Use an action-conditional LAX/RELAX estimator instead of A2C | Reinforcement-learning policies · Lunar Lander | episodes to solve: 162374 ± 17241 -> 68712 ± 20668 | Helped |
| Use LAX instead of A2C | Continuous-action reinforcement-learning policy · Inverted Pendulum | episodes to solve: 6243 ± 164 -> 2067 ± 412 | Helped |
| Optimize the control variate by minimizing the estimator's variance rather than squared prediction error | LAX/RELAX control variate · Black-box gradient estimation | gradient-estimator variance | Helped |
| Make the baseline depend on the sampled action while retaining the unbiased correction | LAX/RELAX reinforcement-learning estimator · Markov decision processes | gradient-estimator variance | Helped |
Closest papers by meaning
- Gradient Estimation for Binary Latent Variables via Gradient Variance Clipping (AAAI 2023 · similarity 0.89)
- Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Gradient Estimators for Reinforcement Learning (NeurIPS 2019 · similarity 0.88)
- Total stochastic gradient algorithms and applications in reinforcement learning (NeurIPS 2018 · similarity 0.88)
- Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States (ICML 2024 · similarity 0.88)
- Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (NeurIPS 2017 · similarity 0.88)
- Biased Gradient Estimate with Drastic Variance Reduction for Meta Reinforcement Learning (ICML 2022 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.