Research map / Reinforcement learning / Policy gradient and actor-critic
Pausing Policy Learning in Non-stationary Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a forecasting online RL framework that forecasts future Q-values from past estimates and deliberately pauses policy updates, showing via dynamic-regret bounds and Mujoco experiments that non-zero policy hold durations improve reward over continual updating.
Problem
Real-time RL in time-varying (non-stationary) environments, where data is collected in the past, policy is updated in the present, and deployed in the future; the question is how often and when to update the policy to minimize dynamic regret.
Approach
Forecast the future Q-function at each update time via linear extrapolation from past Q estimates, optimize the policy toward the forecast during an update window, then hold the policy fixed for a pause window; theory shows a non-zero hold duration can give a sharper dynamic-regret bound under non-stationarity.
Key results
- On HalfCheetah and Swimmer, FSAC (forecasting SAC) yields higher average return than SAC in most cases across update frequencies and 36 hyperparameter settings.
- The best update frequency gamma_f is not 1.0, i.e., continual policy updating does not maximize average reward in these experiments.
- In the tabular goal-switching cliffworld, the forecasting Q method recovers an optimal policy shortly after the switch at step 10000 while the reactive method fails.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use future-Q forecasting instead of reactive Q-learning | Q-learning · Goal switching cliffworld | reward per step | Helped |
| add policy hold periods rather than continually updating | FSAC · HalfCheetah | average reward | Helped |
| add policy hold periods rather than continually updating | FSAC · Swimmer | average reward | Helped |
| set update frequency to γ_f = 1.0 instead of allowing policy hold periods (ablation) | FSAC · HalfCheetah and Swimmer | average reward | Hurt |
| vary prediction length l_f across 5, 15, and 20 (ablation) | FSAC · HalfCheetah and Swimmer | average reward | Mixed |
Surprising findings
- Continuously updating the policy (update ratio 1.0) does not yield the best average reward; intermittent pausing performs better.
- Theory shows that in stationary environments zero hold is optimal, but a positive hold duration can minimize the non-stationarity regret bound in non-stationary ones.
Limitations
- Forecasting error bound depends on uncertainty terms that are not closed-form; the optimal hold ratio (Theorem) lacks a closed-form solution and is solved numerically.
- Assumes one trajectory per unit time and abstract time-to-iteration correspondence, which may not match real systems.
- Mujoco experiments use a self-designed non-stationary variant and report results from only 36 hyperparameter settings with 0.5 std error bars.
Experimental setup
Datasets: Mujoco Swimmer (modified non-stationary) Mujoco HalfCheetah (modified non-stationary) switching goal cliffworld (tabular)
Benchmarks: Mujoco Swimmer (non-stationary variant) Mujoco HalfCheetah (non-stationary variant) switching goal cliffworld
Metrics: average reward per episode reward per step dynamic regret (theoretical)
Baselines: SAC reactive Q-learning
Builds on: Soft Actor-Critic Q-learning natural policy gradient follow-the-meta-leader
Closest papers by meaning
- Enabling Realtime Reinforcement Learning at Scale with Staggered Asynchronous Inference (ICLR 2025 · similarity 0.90)
- Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · similarity 0.90)
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- Online Reinforcement Learning in Non-Stationary Context-Driven Environments (ICLR 2025 · similarity 0.89)
- DARLING: Detection Augmented Reinforcement Learning with Non-Stationary Guarantees (NeurIPS 2026 · similarity 0.89)
- Optimal Dynamic Regret by Transformers for Non-Stationary Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.