Research map / Reinforcement learning / Policy gradient and actor-critic
Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adam-Rel, which resets Adam's timestep to zero at each stationary-objective change (retaining momentum), and shows improved PPO and DQN performance on Craftax-Classic, Atari-57, and Atari-10.
Problem
Adam's bias-corrected update size becomes large after abrupt gradient-magnitude changes caused by nonstationary RL objectives (e.g., target network updates, PPO's per-rollout objective), because the global timestep assumes stationarity.
Approach
Instead of resetting the whole Adam optimizer state at target changes, reset only the timestep t to 0 (local timestep within each stationary objective), keeping momentum estimates. This bounds the update size under gradient increases and reduces to learning-rate annealing/warmup when they do not occur.
Key results
- On Atari-10 with DQN, Adam-Rel achieves 65.7% human-normalized performance vs 28.8% for Adam.
- On Atari-57 with PPO, Adam-Rel outperforms Adam in 33 of 55 games and reaches 338% vs 220% human-normalized score at the 75th percentile.
- On Craftax-Classic-1B, Adam-Rel outperforms Adam on all metrics, with clearly higher score, and Adam-MR performs dramatically worse.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reset Adam's timestep at each new objective instead of using the global timestep | DQN · Atari-10 | human-normalized performance: 28.8% -> 65.7% | Helped |
| Reset all optimizer state at target changes instead of retaining momentum and resetting only the timestep (ablation) | DQN · Atari-10 | human-normalized performance: 65.7% vs. 23.5% | Hurt |
| Reset Adam's timestep at each new objective | PPO · Atari-57, 75th percentile | human-normalized score: 220% -> 338% | Helped |
| Reset all optimizer state when PPO samples a new batch instead of retaining momentum (ablation) | PPO · Craftax-Classic-1B | score | Hurt |
| Set β₁ = β₂ instead of using Adam's default hyperparameters (ablation) | PPO · Craftax-Classic-1B | score | Hurt |
| Set β₁ = β₂ instead of using Adam's default hyperparameters (ablation) | PPO · Atari-57 | performance | No clear effect |
| Use Adam-Rel rather than Adam with Polyak-averaged target updates | DQN · Atari-10 with Polyak averaging | median performance | Helped |
Surprising findings
- Fully resetting the Adam optimizer (Adam-MR) performs substantially worse than plain Adam, contrary to prior findings, suggesting retaining momentum across target changes matters.
- Resetting only the timestep outperforms resetting the whole optimizer, undermining the gradient contamination explanation.
Limitations
- Analysis focuses on abrupt nonstationarity; continuous (Polyak averaging) target updates show weaker gains and no clear method for resetting in that setting.
- Theoretical model assumes a step increase in gradient magnitude, which does not match observed gradient norm dynamics (observed increases were ~29% for Adam and ~45% for Adam-Rel).
- Evaluated only on PPO and DQN, Atari and Craftax; continuous control and other domains untested.
Experimental setup
Datasets: Atari-57 Atari-10 Craftax-Classic
Benchmarks: Atari-57 Atari-10 Craftax-Classic-1B
Metrics: human-normalized score interquartile mean median score (geometric mean of achievement success rates) number of achievements performance profile
Baselines: Adam Adam-MR Adam with beta1=beta2
Builds on: Adam PPO DQN Adam-MR (momentum reset for target changes) RAdam
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.87)
- In Search of Adam’s Secret Sauce (NeurIPS 2025 · similarity 0.86)
- Adam with model exponential moving average is effective for nonconvex optimization (NeurIPS 2024 · similarity 0.86)
- SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Cores (ICLR 2024 · similarity 0.86)
- The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics (NeurIPS 2026 · similarity 0.86)
- Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.