Research map / Reinforcement learning / Policy gradient and actor-critic
Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control
Nathan Rahn, Pierluca D'Oro, Harley Wiltzer, Pierre-Luc Bacon, Marc G Bellemare
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| interpolate linearly between policies from the same run instead of from different runs | TD3 policies on Brax · Brax (hopper, walker2d, ant, halfcheetah), 500 pairs each | Below-Threshold Proportion (return collapses) | Improved |
| interpolate between same-run policies on Atari instead of different-run policies | PPO policies · ALE (QBert, Breakout, MsPacman, Seaquest) | Below-Threshold Proportion | Improved |
| add rejection of updates based on post-update return CVaR (Algorithm 1) on top of TD3 | TD3 · Brax tasks, 40 gradient steps, 5 runs per policy | left-tail probability (LTP) reduction | Improved |
| use 0.1-CVaR as the rejection heuristic rather than other alpha values (ablation) | Algorithm 1 rejection procedure · Brax | robustness vs performance balance | Improved |
| rank policies by 1.0-CVaR (the mean) instead of 0.01-CVaR (ablation) | policy ranking · Brax scatter plots | sensitivity to LTP | Hurt |
| train policies by behavior cloning of a TD3 policy instead of TD3 directly | policies learned by BC · 4 Brax environments, 10 runs x 5 policies | mean LTP | No clear effect |
| clone a TD3 policy with behavior cloning | BC vs TD3 teacher · 4 Brax environments | correlation of mean return and LTP with teacher | Mixed |
| measure post-update LTP from states other than the initial state | TD3 (3 runs) · ant and walker2d | LTP over states | No clear effect |
| zoom in on the landscape plot with a smaller window at same granularity (ablation) | return landscape visualization · hopper SAC policy | neighborhood shape | No clear effect |
| measure post-update return distributions on discrete Atari with PPO | LTP metric · ALE, 5 runs, 10 million steps | descriptive power of LTP: 10 million steps | Mixed |
Closest papers by meaning
- Distributional Policy Optimization: An Alternative Approach for Continuous Control (NeurIPS 2019 · similarity 0.88)
- ContraDiff: Planning Towards High Return States via Contrastive Learning (ICLR 2025 · similarity 0.87)
- Beyond Expected Return: Accounting for Policy Reproducibility When Evaluating Reinforcement Learning Algorithms (AAAI 2024 · similarity 0.86)
- Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- On Corruption-Robustness in Performative Reinforcement Learning (AAAI 2025 · similarity 0.86)
- The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
