AI Paper Atlas

Research map / Reinforcement learning / Policy gradient and actor-critic

Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control

Nathan Rahn, Pierluca D'Oro, Harley Wiltzer, Pierre-Luc Bacon, Marc G Bellemare

NeurIPS 2023 · Reinforcement learning › Policy gradient and actor-critic · approach: Regularized scalable deep value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
interpolate linearly between policies from the same run instead of from different runsTD3 policies on Brax · Brax (hopper, walker2d, ant, halfcheetah), 500 pairs eachBelow-Threshold Proportion (return collapses)Improved
interpolate between same-run policies on Atari instead of different-run policiesPPO policies · ALE (QBert, Breakout, MsPacman, Seaquest)Below-Threshold ProportionImproved
add rejection of updates based on post-update return CVaR (Algorithm 1) on top of TD3TD3 · Brax tasks, 40 gradient steps, 5 runs per policyleft-tail probability (LTP) reductionImproved
use 0.1-CVaR as the rejection heuristic rather than other alpha values (ablation)Algorithm 1 rejection procedure · Braxrobustness vs performance balanceImproved
rank policies by 1.0-CVaR (the mean) instead of 0.01-CVaR (ablation)policy ranking · Brax scatter plotssensitivity to LTPHurt
train policies by behavior cloning of a TD3 policy instead of TD3 directlypolicies learned by BC · 4 Brax environments, 10 runs x 5 policiesmean LTPNo clear effect
clone a TD3 policy with behavior cloningBC vs TD3 teacher · 4 Brax environmentscorrelation of mean return and LTP with teacherMixed
measure post-update LTP from states other than the initial stateTD3 (3 runs) · ant and walker2dLTP over statesNo clear effect
zoom in on the landscape plot with a smaller window at same granularity (ablation)return landscape visualization · hopper SAC policyneighborhood shapeNo clear effect
measure post-update return distributions on discrete Atari with PPOLTP metric · ALE, 5 runs, 10 million stepsdescriptive power of LTP: 10 million stepsMixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy