Research map / Reinforcement learning / MDPs and planning
Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RESeL, a recurrent off-policy RL method that uses a lower learning rate for the RNN context encoder than for MLP layers, built on SAC with REDQ-style ensemble critics and Mamba encoders, and reports gains on 18 POMDP tasks and 5 MuJoCo tasks.
Problem
Recurrent RL for POMDPs suffers training instability because RNN output variations are amplified over long sequences, causing unstable policies or divergence.
Approach
Theoretically bounds how one-step parameter updates amplify RNN output differences over time, then uses a context-encoder-specific (smaller) learning rate for the RNN context encoder while keeping the normal learning rate for MLP layers, integrated into SAC with REDQ ensemble critics.
Key results
- On the classic MuJoCo tasks at 5M steps, RESeL averages 9532 versus TD7's 10010 (about 4.8% lower) and improves over SAC by about 34.2% on average.
- In the dynamics-randomized tasks, RESeL shows significant improvement over other methods in Ant, Humanoid, and Walker2d, and performs on par with ESCP in Hopper.
- Ablation: with a uniform learning rate of 3e-4 for all parts, performance drops significantly, and some AntBLT-V and HalfCheetahBLT-V runs diverged to NaN or infinite outputs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a lower learning rate for the context encoder while keeping the other layers at 3e-4 (ablation) | RESeL recurrent policy · Classic POMDP tasks | final return | Helped |
| Set the context-encoder learning rate equal to the other-layer learning rate at 3e-4 (ablation) | RESeL recurrent policy · AntBLT-V and HalfCheetahBLT-V | training stability | Hurt |
| Vary the context-encoder learning rate with other-layer learning rate fixed at 3e-4 (ablation) | RESeL · WalkerBLT-V | final return | Mixed |
| Vary the other-layer learning rate with context-encoder learning rate fixed at 1e-5 (ablation) | RESeL · WalkerBLT-V | final return | Mixed |
| Replace the Mamba context encoder with a GRU context encoder (ablation) | RESeL · HalfCheetahBLT-V and WalkerBLT-V | return | Helped |
| Replace the Mamba context encoder with a GRU context encoder (ablation) | RESeL · HalfCheetahBLT-P | return | Hurt |
| Use RESeL instead of SAC | SAC · Five MuJoCo locomotion tasks | average return: +34.2% | Helped |
| Use RESeL instead of TD7 | TD7 · Five MuJoCo locomotion tasks | average return: 4.8% average performance drop | Hurt |
| Use RESeL instead of TD7 | TD7 · Hopper and Walker | return | Helped |
| Use RESeL instead of previous recurrent RL baselines | MF-RNN, PPO-GRU, and A2C-GRU · Classic POMDP tasks | return | Helped |
| Use RESeL instead of MF-RNN | MF-RNN · HalfCheetahBLT-P | return | No clear effect |
| Use RESeL instead of MF-GPT and MF-RNN | MF-GPT and MF-RNN · Key-to-Door, credit assignment lengths 250 and 500 | success rate | Helped |
Surprising findings
- The output variation induced by a one-step update grows about 400% by step 20 in the RNN, and a uniform learning rate would need to be about twentyfold larger to match it.
- Under the Key-to-Door credit assignment tasks, RESeL outperforms the Transformer-based MF-GPT on the harder lengths 250 and 500.
Limitations
- Unclear how the RNN context encoder helps performance in MDP tasks.
- Introduces an extra hyperparameter LR_CE whose relation to LR_other is not explored; tuning is needed.
- GRU variant was tested only for 0.75M steps on four tasks without hyperparameter tuning.
Experimental setup
Datasets: PyBullet locomotion POMDP tasks (AntBLT, HalfCheetahBLT, HopperBLT, WalkerBLT) MuJoCo gravity-randomized tasks AntDir, CheetahDir, HalfCheetahVel, Wind meta-RL tasks Key-to-Door MuJoCo v2 locomotion tasks
Benchmarks: AntBLT-V AntBLT-P HalfCheetahBLT-V HalfCheetahBLT-P HopperBLT-V HopperBLT-P WalkerBLT-V WalkerBLT-P MuJoCo gravity-randomized tasks AntDir CheetahDir HalfCheetahVel
Metrics: average return success rate training stability (NaN/infinite outputs) output variation
Baselines: MF-RNN VRM GPIDE ESCP EPI OSI ProMP PEARL RL2 VariBad MF-GPT TD3
Builds on: SAC REDQ TD3 MF-RNN Mamba
Closest papers by meaning
- Recurrent Predictive State Policy Networks (ICML 2018 · similarity 0.91)
- Are Expressive Models Truly Necessary for Offline RL? (AAAI 2025 · similarity 0.89)
- Distribution-Adaptive Policy Optimization (NeurIPS 2026 · similarity 0.89)
- Rethinking Transformers in Solving POMDPs (ICML 2024 · similarity 0.89)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.88)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.