Research map / Reinforcement learning / Visual and representation RL
Learning to Reuse Policies in State Evolvable Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes state evolvable RL (SERL) and proposes Lapse, which reuses an old robust policy via a GAN-based state reconstruction model and an offline-RL adaptive policy, combined by an automatically adjusted ensemble weight.
Problem
Deployed RL policies fail when sensor-derived state features evolve (sensors vanish or are added), and adapting by online trial-and-error is costly.
Approach
Reuse the old policy by composing it with a learned state reconstruction model (conditional GAN plus L_p loss) after training it robustly (action-distribution smoothness), and train a new policy offline from limited old-space experience to exploit new sensors; aggregate both with an automatically adjusted ensemble weight based on returns and policy divergences.
Key results
- Lapse achieves an overall normalized return of 91.09 versus 45.51 for the strongest baseline (Offline), roughly 2x.
- On HalfCheetah Lapse scores 82.56 against 28.31 for PAD, and on Pong it reaches 100.0.
- Ablations on Pong show that removing the policy aggregation (W/o kappa) or offline component (W/o offline) degrades performance in later stages.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Lapse instead of Offline | Ant adaptation · Ant | normalized average return: 82.63 -> 84.02 | Helped |
| apply Lapse instead of PAD | HalfCheetah adaptation · HalfCheetah | normalized average return: 28.31 -> 82.56 | Helped |
| apply Lapse instead of FPT | Hopper adaptation · Hopper | normalized average return: 78.94 -> 98.89 | Helped |
| apply Lapse instead of FPT | Walker adaptation · Walker | normalized average return: 19.94 -> 83.85 | Helped |
| apply Lapse instead of Offline | BankHeist adaptation · BankHeist | normalized average return: 61.07 -> 95.34 | Helped |
| apply Lapse instead of PAD | Freeway adaptation · Freeway | normalized average return: 78.86 -> 98.32 | Helped |
| apply Lapse instead of Offline | Pong adaptation · Pong | normalized average return: 33.56 -> 100.0 | Helped |
| apply Lapse instead of PAD | RoadRunner adaptation · RoadRunner | normalized average return: 28.67 -> 85.65 | Helped |
| use a static ensemble weight κn+1 = 0.5 instead of automatic adjustment (ablation) | Lapse · Pong | normalized average return | Hurt |
| remove the offline adaptive policy (ablation) | Lapse · Pong | normalized average return | Hurt |
| remove the reconstruction policy (ablation) | Lapse · Pong | normalized average return | Hurt |
| remove robustness regularization (ablation) | Lapse · Pong | normalized average return | Hurt |
Surprising findings
- Offline RL alone is on par with Lapse on Ant and Hopper, and RL-GAN and LUSR perform poorly despite their domain-adaptation design.
- Even with an optimal reconstruction model, the reconstruction policy fails to surpass the old policy, indicating compounding error.
Limitations
- Requires a dataset of paired old and new state features collected during a short overlap period (prior knowledge of which sensors change).
- May be less effective under drastic state space evolutions.
- Reconstruction error accumulates over repeated evolutions.
Experimental setup
Datasets: MuJoCo Ant MuJoCo HalfCheetah MuJoCo Hopper MuJoCo Walker Atari BankHeist Atari Freeway Atari Pong Atari RoadRunner
Benchmarks: MuJoCo Ant MuJoCo HalfCheetah MuJoCo Hopper MuJoCo Walker Atari BankHeist Atari Freeway Atari Pong Atari RoadRunner
Metrics: normalized average return across evolving stages
Baselines: RL-GAN LUSR PAD Offline RL FPT CUP
Builds on: TD3+BC RORL RADIAL-DQN Wocar-PPO CUP
Closest papers by meaning
- Efficient Test-Time Adaptation For Robot Policies (NeurIPS 2026 · similarity 0.88)
- From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
- Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancy (NeurIPS 2024 · similarity 0.87)
- Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias (NeurIPS 2026 · similarity 0.87)
- When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
- Blending Imitation and Reinforcement Learning for Robust Policy Improvement (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.