Research map / Reinforcement learning / Visual and representation RL
Discovering Temporally-Aware Reinforcement Learning Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Augments learned policy gradient (LPG) and learned policy optimization (LPO) with training-lifetime conditioning and meta-optimizes them with evolution strategies, yielding temporally-aware objectives that adapt over training and generalize to unseen horizons and environments.
Problem
Existing meta-learned RL objective functions ignore the total training horizon, and meta-gradient methods with truncated unrolls are myopic, so they fail to learn lifetime-adaptive updates.
Approach
Condition learned objectives on the agent's lifetime fraction n/N and log training horizon, and meta-optimize with evolution strategies over full agent lifetimes (with antithetic task sampling for multi-task ES), so the learned loss can adapt its exploration/exploitation schedule.
Key results
- TA-LPG achieves consistently higher final return than LPG across training horizons, reaching maximum performance on the sparse task in 1/8 of LPG's training steps.
- TA-LPO, meta-trained only on MinAtar SpaceInvaders, significantly outperforms LPO and PPO on a range of MinAtar and Brax tasks.
- Meta-gradient TA-LPG achieves lower return at all horizons and shows no horizon adaptation, while ES-trained TA-LPG adapts entropy and update norm to horizon.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add lifetime and training-horizon inputs to LPG and optimize with evolution strategies | LPG · Held-out Grid-Worlds across training horizons | final return: 1/8 of the training steps required for LPG | Helped |
| Add lifetime and training-horizon inputs to LPG and optimize with evolution strategies | LPG · Minigrid across training horizons | final return | Helped |
| Add lifetime conditioning to LPO's drift function | LPO · MinAtar | evaluation return | Helped |
| Add lifetime conditioning to LPO's drift function | LPO · Brax | evaluation return | Helped |
| Replace evolution strategies with meta-gradient optimization (ablation) | TA-LPG · Random Grid-Worlds across training horizons | final return | Hurt |
| Replace evolution strategies with meta-gradient optimization (ablation) | TA-LPG · Random Grid-Worlds across training horizons | horizon adaptation | Hurt |
| Add a heuristic learning-rate schedule (ablation) | LPG · Held-out Grid-Worlds across training horizons | final return | No clear effect |
| Add a heuristic learning-rate schedule (ablation) | LPG · Held-out Grid-Worlds across training horizons | final return | Hurt |
Surprising findings
- Meta-gradient optimization with truncated unrolls fails to learn temporally-adaptive updates, while evolution strategies do.
- A heuristic learning rate schedule applied to LPG has an insignificant effect on its performance.
Limitations
- LPG offers limited theoretical robustness guarantees.
- LPO meta-learns in a more restricted space and may be less expressive.
- Only temporal information is used for conditioning; other information such as optimizer statistics is left for future work.
Experimental setup
Datasets: Grid-World distribution (LPG meta-training) MinAtar SpaceInvaders (LPO meta-training)
Benchmarks: held-out Grid-Worlds Minigrid MinAtar Brax
Metrics: normalized final return policy entropy policy update norm
Baselines: LPG LPO PPO meta-gradient TA-LPG LPG with learning rate schedule
Builds on: Learned Policy Gradient Learned Policy Optimization Mirror Learning PPO PureJaxRL
Closest papers by meaning
- Meta-Gradient Reinforcement Learning with an Objective Discovered Online (NeurIPS 2020 · similarity 0.91)
- Can Learned Optimization Make Reinforcement Learning Less Difficult? (NeurIPS 2024 · similarity 0.90)
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Discovered Policy Optimisation (NeurIPS 2022 · similarity 0.89)
- Improving Generalization in Meta Reinforcement Learning using Learned Objectives (ICLR 2020 · similarity 0.89)
- First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.