Research map / Reinforcement learning / Offline RL
Online Reinforcement Learning in Non-Stationary Context-Driven Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LCPO, an on-policy RL method that mitigates catastrophic forgetting in context-driven non-stationary environments by constraining policy updates via KL on out-of-distribution past experiences, without task labels.
Problem
Online RL in non-stationary, context-driven environments suffers catastrophic forgetting when the exogenous context changes; existing fixes need task labels, brittle heuristics, or unstable off-policy methods.
Approach
Anchor the policy on old experiences whose context is out-of-distribution relative to the current batch by adding a KL constraint on those samples to a TRPO-style constrained policy gradient update, using a bounded reservoir-sampled buffer and an OOD detector instead of task labels.
Key results
- LCPO outperforms all online baselines across gymnasium environments and contexts and remains closest to the best offline prescient agent in normalized return.
- On Pendulum-v1 wind traces, LCPO achieves -190, -355, -240, -378 lifelong return versus A2C's -201, -377, -307, -399 and MBPO's -325 to -843.
- LCPO maintains performance with buffer size as small as n_b=500 but degrades below that; LCPO is about 1.5x the wallclock cost of A2C.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LCPO instead of MBPO | Pendulum-v1 online learning · Context Trace 1 | lifelong return: -325 -> -190 | Helped |
| Use LCPO instead of MBPO | Pendulum-v1 online learning · Context Trace 2 | lifelong return: -843 -> -355 | Helped |
| Use LCPO instead of MBPO | Pendulum-v1 online learning · Context Trace 3 | lifelong return: -603 -> -240 | Helped |
| Use LCPO instead of MBPO | Pendulum-v1 online learning · Context Trace 4 | lifelong return: -553 -> -378 | Helped |
| Provide MBPO with the ground-truth environment model (ablation) | MBPO on Pendulum-v1 · Context Traces 1–4 | lifelong return: -325 vs -320; -843 vs -870; -603 vs -637; -553 vs -555 | No clear effect |
| Use LCPO rather than A2C | online RL in straggler mitigation · Workload 1 | tail latency: 1716 -> 1048 | Helped |
| Use LCPO rather than A2C | online RL in straggler mitigation · Workload 2 | tail latency: 604 vs 586 | No clear effect |
| Vary the OOD threshold (ablation) | LCPO on gym environments · Gymnasium contexts | normalized return | Mixed |
| Use Mahalanobis distance instead of L2 distance for OOD sampling (ablation) | LCPO on gym environments · Gymnasium contexts | normalized return | Helped |
| Vary the OOD threshold among the reported settings (ablation) | LCPO on straggler mitigation · Workloads 1–2 | normalized return | No clear effect |
| Reduce the experience buffer below 500 samples (ablation) | LCPO on gym environments · Gymnasium contexts | normalized return | Hurt |
| Use a buffer of 500 samples or more (ablation) | LCPO on gym environments · Gymnasium contexts | normalized return | Helped |
Surprising findings
- Ideal MBPO with ground-truth dynamics performs similarly to learned-model MBPO, indicating the off-policy learning procedure rather than model accuracy causes its poor performance.
- Off-policy baselines SAC and DDQN struggle to outperform on-policy A2C in the online setting, and A2C is the strongest baseline despite not being designed for non-stationary RL.
Limitations
- LCPO performance is sensitive to the OOD threshold sigma and requires tuning an OOD detector.
- Buffer sizes below about 500 samples degrade performance; high-dimensional contexts may need larger buffers.
- Exploration uses automatic entropy tuning not designed for non-stationary settings; no latent-context experiments.
Experimental setup
Datasets: Gymnasium Pendulum-v1 with synthetic wind contexts InvertedPendulum-v4 InvertedDoublePendulum-v4 Hopper-v4 Reacher-v4 straggler mitigation workloads from Microsoft production cluster
Benchmarks: Gymnasium Pendulum-v1 (wind contexts) InvertedPendulum-v4 InvertedDoublePendulum-v4 Hopper-v4 Reacher-v4 straggler mitigation
Metrics: normalized lifelong return tail latency (negative reward) 95% confidence intervals
Baselines: Online EWC Sliding OGD Benna-Fusi DQN MBCD MBPO CLEAR PT-DQN SAC DDQN A2C TRPO Prescient agent
Builds on: TRPO Advantage actor-critic (A2C) Generalized advantage estimation Reservoir sampling Soft actor-critic
Closest papers by meaning
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
- Pausing Policy Learning in Non-stationary Reinforcement Learning (ICML 2024 · similarity 0.89)
- Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · similarity 0.89)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- The Power of Resets in Online Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.