AI Paper Atlas

Research map / Optimization / Optimizers

Batch size-invariance for policy optimization

Jacob Hilton, Karl Cobbe, John Schulman

NeurIPS 2022 · Optimization › Optimizers · approach: Memory-efficient LLM optimizers · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Decouple the proximal policy from the behavior policy and use the recent policy as proximalPPO with stale data · Procgen Benchmark, artificial stalenessperformance under stalenessImproved
Use a proximal policy EWMA and adjust hyperparameters for batch sizePPG-EWMA · Procgen Benchmark, 16 environmentsdifference in final mean normalized return across batch sizes: 0.052; 0.019 excluding HeistImproved
Add an EWMA proximal policyPPO and PPG · Procgen Benchmark, last 8 environmentsnormalized returnImproved
Remove the Adam step size adjustment (ablation)PPG-EWMA · Procgen Benchmark, smaller batch sizesbatch size-invariance: 0.709Hurt
Remove the advantage normalization adjustment (ablation)PPG-EWMA · Procgen Benchmark, smallest batch sizesbatch size-invariance: 0.224Hurt
Remove the advantage normalization adjustment (ablation)PPG-EWMA · Procgen Benchmark, Default vs 4difference-of-differences in final normalized return: -0.014No clear effect
Remove the EWMA decay-rate adjustment (ablation)PPG-EWMA · Procgen Benchmark, different batch sizesbatch size-invariance: 0.041Hurt
Remove EWMA and use PPG (ablation)PPG-EWMA · Procgen Benchmark, different batch sizesbatch size-invariance: 0.030Hurt
Use a linear instead of square root Adam step size adjustment (ablation)PPG-EWMA · Procgen Benchmark, different batch sizesbatch size-invarianceHurt
Adjust Adam's beta_2 only, or both beta_1 and beta_2 (ablation)PPG-EWMA · Heistdifference in performance across batch sizes: 0.55No clear effect
Tune the proximal EWMA center of mass and KL penalty together by halving the former and doubling the latter (ablation)PPG-EWMA · StarPilotnormalized returnMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy