Research map / Reinforcement learning / Policy gradient and actor-critic
Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TRAC, a parameter-free optimizer wrapper built on online convex optimization that adaptively regularizes policy weights toward a reference point, mitigating loss of plasticity in lifelong PPO across Procgen, Atari, and Gym Control.
Problem
Lifelong reinforcement learning suffers from loss of plasticity under distribution shifts, and existing remedies (L2 regularization, resetting) require environment-dependent hyperparameters that cannot be tuned in a one-shot setting.
Approach
Wrap any base optimizer (e.g., Adam) with a parameter-free OCO tuner (erfi-based 1D tuner aggregated over discount factors) that scales the update toward a reference point online, acting as a data-dependent L2-like regularizer without tuning.
Key results
- On Procgen, TRAC PPO reaches cumulative reward 12.5 on Starpilot vs 3.4 for Adam PPO and 3.6 for CReLU (x10^5), with 3,212.42% average normalized improvement over Adam PPO.
- On Atari 6 and 9 action-space settings, TRAC PPO averages 329.73% normalized improvement over Adam PPO and 68.71% over CReLU.
- On Gym Control, TRAC PPO averages 204.18% improvement over Adam PPO and 1044.24% over CReLU; CartPole cumulative reward 39.6 vs 5.1 for Adam PPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TRAC instead of Adam | PPO · Procgen (Starpilot, Dodgeball, Chaser, Fruitbot) | cumulative mean episode reward: 3,212.42% | Helped |
| use TRAC instead of CReLU | PPO · Procgen (Starpilot, Dodgeball, Chaser, Fruitbot) | cumulative mean episode reward: 120.88% | Helped |
| use TRAC instead of Adam | PPO · Atari 6 and Atari 9 | cumulative mean episode reward: 329.73% | Helped |
| use TRAC instead of CReLU | PPO · Atari 6 and Atari 9 | cumulative mean episode reward: 68.71% | Helped |
| use TRAC instead of Adam | PPO · Gym Control (CartPole, Acrobot, LunarLander) | cumulative mean episode reward: 204.18% | Helped |
| use TRAC instead of CReLU | PPO · Gym Control (CartPole, Acrobot, LunarLander) | cumulative mean episode reward: 1044.24% | Helped |
| replace Adam with TRAC | IMPALA · Procgen | average normalized reward: 21.83% | Helped |
| replace Adam with TRAC | Online EWC · Procgen | average normalized reward: 15.86% | Helped |
| replace Adam with TRAC | CLEAR · Procgen | average normalized reward: 14.41% | Helped |
| replace Adam with TRAC | Modulating Masks · Procgen | average normalized reward: 10.14% | Helped |
| replace TRAC with Mechanic (ablation) | PPO · Gym Control | mean reward: 0.6018 -> 0.5755; 0.3518 -> 0.3008; 0.7044 -> 0.6396 | Hurt |
| warmstart with Adam before using TRAC (ablation) | TRAC PPO · Atari | reward | No clear effect |
Surprising findings
- A random initialization serves as a good enough reference point for TRAC even when tasks are similar, and warmstarting is not essential.
- TRAC outperforms privileged weight-resetting, which resets parameters at every task boundary, suggesting positive transfer.
Limitations
- TRAC can underperform the baseline optimizer at the start of deployment.
- Warmstarting benefits are environment-specific and not consistent.
- Theoretical justification relies on convex OCO while lifelong RL is nonconvex and nonstationary.
Experimental setup
Datasets: Procgen Atari 2600 (ALE) CartPole-v1 Acrobot-v1 LunarLander-v2
Benchmarks: Procgen Atari Gym Control
Metrics: cumulative sum of mean episode reward normalized improvement
Baselines: Adam PPO CReLU IMPALA Online EWC CLEAR Modulating Masks Mechanic L2 regularization layer normalization plasticity injection privileged weight resetting
Builds on: PPO Adam Mechanic CReLU Online EWC
Closest papers by meaning
- Online Reinforcement Learning in Non-Stationary Context-Driven Environments (ICLR 2025 · similarity 0.87)
- TD-MPC2: Scalable, Robust World Models for Continuous Control (ICLR 2024 · similarity 0.87)
- SOMBRL: Scalable and Optimistic Model-Based RL (NeurIPS 2025 · similarity 0.87)
- DARLING: Detection Augmented Reinforcement Learning with Non-Stationary Guarantees (NeurIPS 2026 · similarity 0.86)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- The Power of Resets in Online Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.