Research map / Reinforcement learning / Visual and representation RL
Position: Lifetime tuning is incompatible with continual reinforcement learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that lifetime hyperparameter tuning is incompatible with continual RL, and proposing k-percent tuning (tune only on a small fraction of the agent's lifetime) with experiments on DQN and SAC across non-stationary Catch, Cartpole, Quadruped walk-to-run, and Jelly Bean World.
Problem
Standard RL evaluation tunes hyperparameters over the agent's entire lifetime, which hides the loss of plasticity and non-stationarity that continual RL benchmarks are meant to test, so reported continual RL progress is misleading.
Approach
Constrain hyperparameter tuning to a small fixed percentage k of the total lifetime (k-percent tuning), then deploy the chosen configuration unchanged for the full run, so that continual learning ability rather than lifetime-specific tuning is measured.
Key results
- In Non-stationary Catch, DQN with lifetime tuning performs well, but with 1% tuning its performance collapses, while W0-DQN remains stable over the full lifetime.
- In Jelly Bean World, with 10% tuning W0-DQN maintains performance while DQN and PT-DQN collapse over time; with lifetime tuning all three algorithms look similar.
- In quadruped walk-to-run with SAC under 1% tuning, most mitigations (e.g., CReLU, weight normalization) improve over SAC baseline, while W0 regularization decreases performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Limit DQN hyperparameter tuning to 1% of the lifetime instead of tuning for the full lifetime | DQN · Continuing Cartpole | performance | Hurt |
| Tune DQN and W0-DQN for the full lifetime | DQN and W0-DQN · Non-stationary Catch | performance | No clear effect |
| Add W0 regularization to DQN (ablation) | DQN · Non-stationary Catch, 20× longer experiment | performance | Helped |
| Add W0 regularization, L2 regularization, CReLU, PT-DQN, or weight normalization to DQN (ablation) | DQN · Non-stationary Catch, 1% tuning | performance | Helped |
| Add Layer Normalization to DQN (ablation) | DQN · Non-stationary Catch, 1% tuning | performance | Hurt |
| Add Layer Normalization to DQN (ablation) | DQN · Continuing Cartpole, 1% tuning | performance | Helped |
| Add PT-DQN to DQN (ablation) | DQN · Continuing Cartpole, 1% tuning | performance | No clear effect |
| Add W0 regularization, L2 regularization, CReLU, or weight normalization to DQN (ablation) | DQN · Continuing Cartpole, 1% tuning | performance | Mixed |
| Use hyperparameters tuned on 1% of quadruped-walk rather than default SAC hyperparameters | SAC · Quadruped-walk-to-run | performance | Helped |
| Add W0 regularization to SAC (ablation) | SAC · Quadruped-walk-to-run, 1% tuning | performance | Hurt |
| Add CReLU to SAC (ablation) | SAC · Quadruped-walk-to-run, 1% tuning | performance | Helped |
| Combine CReLU with weight normalization in SAC (ablation) | SAC · Quadruped-walk-to-run, 1% tuning | performance | Helped |
Surprising findings
- Under lifetime tuning, DQN performs comparably to continual-learning mitigations in non-stationary environments, so lifetime tuning hides differences between algorithms.
- Under k-percent tuning, the learning rate selected for SAC on quadruped walk-run was larger than the default, which performs well short-term but fails over the long run.
Limitations
- k-percent tuning is not a faithful model of real deployment, since it assumes multiple independent trials are possible
- The proposed protocol is a simple starting point and the authors expect it to be improved or replaced
- Experiments are on small toy and classic-control environments with a limited set of algorithms and mitigations
Experimental setup
Datasets: Non-stationary Catch Continuing Cartpole DeepMind Control Suite quadruped walk-to-run Jelly Bean World
Benchmarks: Non-stationary Catch Continuing Cartpole DeepMind Control Suite quadruped walk-to-run Jelly Bean World
Metrics: area under the curve average reward performance over time
Baselines: DQN SAC W0-DQN L2 regularization CReLU PT-DQN Weight normalization Layer normalization
Builds on: DQN Soft Actor-Critic W0 regularization PT-DQN Layer normalization
Closest papers by meaning
- Continual Learning In Environments With Polynomial Mixing Times (NeurIPS 2022 · similarity 0.87)
- Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- On-Policy Algorithms for Continual Reinforcement Learning (Student Abstract) (AAAI 2025 · similarity 0.86)
- Continual Reinforcement Learning by Planning with Online World Models (ICML 2025 · similarity 0.85)
- A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.