Research map / Reinforcement learning / Visual and representation RL
Can Learned Optimization Make Reinforcement Learning Less Difficult?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Meta-learns a black-box update rule (OPEN) via OpenAI evolution strategies, conditioned on RL-specific features (training/batch proportion, layer proportion, dormancy) with learnable parameter-space noise, to replace Adam for PPO training.
Problem
Standard optimizers like Adam are ill-suited to RL's non-stationarity, plasticity loss, and exploration needs, and prior learned optimizers transfer poorly to RL.
Approach
A learned optimizer whose inputs include RL-targeted signals (two training timescales, layer proportion, neuron dormancy) and whose output includes a learnable per-parameter noise term applied to the actor for exploration, meta-trained with OpenAI ES on final return, with zero-mean updates for stability.
Key results
- In single-task training on three MinAtar environments, OPEN significantly outperforms all baselines, including Adam, Lion, VeLO and Optim4RL.
- In multi-task MinAtar training, OPEN is the only learned optimizer with aggregate Adam-normalized score above Adam.
- On gridworld generalization, OPEN outperforms tuned Adam in every out-of-support environment and agent size tested, and transfers 0-shot to Craftax-Classic better than Adam (0-shot).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OPEN instead of the baselines | PPO agents trained separately on MinAtar environments · MinAtar single-task | IQM final return | Helped |
| Use OPEN instead of the baselines | PPO agents trained across MinAtar environments · MinAtar multi-task | Adam-normalized aggregate score | Helped |
| Use OPEN instead of Adam | PPO agents · Gridworlds, in-distribution and out-of-support | Adam-normalized IQM return | Helped |
| Transfer the gridworld-trained optimizer without further training | OPEN · Craftax-Classic | mean return | Helped |
| Add training proportion as an optimizer input (ablation) | OPEN · Breakout | mean final return | Helped |
| Add batch proportion as an optimizer input (ablation) | OPEN · Breakout | mean final return | Helped |
| Add dormancy as an optimizer input (ablation) | OPEN · Breakout | mean final return | Helped |
| Add layer proportion as an optimizer input (ablation) | OPEN · Breakout | mean final return | Helped |
| Add learnable stochasticity to the actor update (ablation) | OPEN · Breakout | mean final return | Helped |
| Add learnable stochasticity to the actor update (ablation) | Separated actor-critic OPEN · Deep Sea, larger environments | return | Helped |
| Add learnable stochasticity to the actor update (ablation) | Separated actor-critic OPEN · Deep Sea, smaller environments | return | Hurt |
| Train OPEN instead of Adam | PQN agents · Asterix | IQM final return | Helped |
Surprising findings
- Naively applying OPEN to Deep Sea struggles, so the actor and critic needed separate updates for exploration to work.
- Optim4RL performed better in multi-task than single-task training, which the authors attribute to more meta-training samples.
Limitations
- Meta-training uses a single random seed and no ES hyperparameter tuning due to compute constraints.
- Normalizing returns by Adam in multi-task training biases learning toward environments where Adam underperforms.
- Learned stochasticity alone hurt performance in smaller Deep Sea environments.
Experimental setup
Datasets: MinAtar Brax Ant gridworld distributions (LPG rand_dense, rand_sparse, rand_long) Minigrid mazes Craftax-Classic bsuite Deep Sea
Benchmarks: MinAtar (Breakout, Asterix, Space Invaders, Freeway) Brax Ant gridworld in-distribution and out-of-support Minigrid mazes Craftax-Classic bsuite Deep Sea
Metrics: interquartile mean final return mean normalized return optimality gap dormancy
Baselines: Adam RMSProp Lion VeLO Optim4RL No Features
Builds on: OpenAI ES Optim4RL VeLO Lion PureJaxRL
Closest papers by meaning
- Meta-Gradient Reinforcement Learning with an Objective Discovered Online (NeurIPS 2020 · similarity 0.90)
- Guided Meta-Policy Search (NeurIPS 2019 · similarity 0.90)
- Discovered Policy Optimisation (NeurIPS 2022 · similarity 0.90)
- Discovering Temporally-Aware Reinforcement Learning Algorithms (ICLR 2024 · similarity 0.90)
- Enhanced Meta Reinforcement Learning via Demonstrations in Sparse Reward Environments (NeurIPS 2022 · similarity 0.90)
- Meta-Reinforcement Learning with Universal Policy Adaptation: Provable Near-Optimality under All-task Optimum Comparator (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.