Research map / Optimization / Optimizers
Batch size-invariance for policy optimization
Jacob Hilton, Karl Cobbe, John Schulman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Decouple the proximal policy from the behavior policy and use the recent policy as proximal | PPO with stale data · Procgen Benchmark, artificial staleness | performance under staleness | Improved |
| Use a proximal policy EWMA and adjust hyperparameters for batch size | PPG-EWMA · Procgen Benchmark, 16 environments | difference in final mean normalized return across batch sizes: 0.052; 0.019 excluding Heist | Improved |
| Add an EWMA proximal policy | PPO and PPG · Procgen Benchmark, last 8 environments | normalized return | Improved |
| Remove the Adam step size adjustment (ablation) | PPG-EWMA · Procgen Benchmark, smaller batch sizes | batch size-invariance: 0.709 | Hurt |
| Remove the advantage normalization adjustment (ablation) | PPG-EWMA · Procgen Benchmark, smallest batch sizes | batch size-invariance: 0.224 | Hurt |
| Remove the advantage normalization adjustment (ablation) | PPG-EWMA · Procgen Benchmark, Default vs 4 | difference-of-differences in final normalized return: -0.014 | No clear effect |
| Remove the EWMA decay-rate adjustment (ablation) | PPG-EWMA · Procgen Benchmark, different batch sizes | batch size-invariance: 0.041 | Hurt |
| Remove EWMA and use PPG (ablation) | PPG-EWMA · Procgen Benchmark, different batch sizes | batch size-invariance: 0.030 | Hurt |
| Use a linear instead of square root Adam step size adjustment (ablation) | PPG-EWMA · Procgen Benchmark, different batch sizes | batch size-invariance | Hurt |
| Adjust Adam's beta_2 only, or both beta_1 and beta_2 (ablation) | PPG-EWMA · Heist | difference in performance across batch sizes: 0.55 | No clear effect |
| Tune the proximal EWMA center of mass and KL penalty together by halving the former and doubling the latter (ablation) | PPG-EWMA · StarPilot | normalized return | Mixed |
Closest papers by meaning
- Adaptive Batch Size for Safe Policy Gradients (NeurIPS 2017 · similarity 0.88)
- On the Optimality of Batch Policy Optimization Algorithms (ICML 2021 · similarity 0.87)
- Model Selection in Batch Policy Optimization (ICML 2022 · similarity 0.87)
- History-Gradient Aided Batch Size Adaptation for Variance Reduced Algorithms (ICML 2020 · similarity 0.87)
- A Parametric Class of Approximate Gradient Updates for Policy Optimization (ICML 2022 · similarity 0.87)
- Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimisation (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
