Research map / Reinforcement learning / Policy gradient and actor-critic
Dr Jekyll & Mr Hyde: the strange case of off-policy policy updates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add an off-policy component using a uniform state distribution to policy updates | Policy updates · Chain domain | normalized performance | Helped |
| Use on-policy updates only (ablation) | Policy updates · Chain domain | normalized performance | Hurt |
| Use constant off-policiness o_t=0.5 instead of relying on the weaker asymptotic coverage condition (ablation) | Policy updates · Chain domain | convergence speed | Helped |
| Use purely uniform updates (o_t=1) instead of a mixture of on-policy and off-policy updates (ablation) | Policy updates · Random MDPs | training speed | Hurt |
| Mix on-policy and off-policy components in the update density | Policy updates · Random MDPs | normalized performance | Helped |
| Add policy entropy regularization (ablation) | Policy updates · Chain domain | convergence to optimal policy | No clear effect |
| Add policy entropy regularization (ablation) | Policy updates · Random MDPs | normalized performance | Hurt |
| Use on-policy updates with a strong UCB bonus for the critic | Actor-critic · Chain domain, size 10, β=0.8 | ability to solve task | Helped |
| Use Jekyll & Hyde with off-policiness o_t=0.5 | Jekyll & Hyde · Chain domain | time to identify optimal policy | Helped |
| Use exploration schedule ε_t=100/t instead of ε_t=10/√t (ablation) | Jekyll & Hyde · Chain domain | asymptotic convergence speed | Helped |
| Use a high UCB critic bonus ν∈{0.1,1} (ablation) | Actor-critic · Random MDPs | performance within experimental time | Hurt |
| Add RND exploration rewards | DDQN and SAC · Four Rooms level 1 | score | Helped |
Closest papers by meaning
- A Parametric Class of Approximate Gradient Updates for Policy Optimization (ICML 2022 · similarity 0.90)
- $\phi$-Update: A Class of Policy Update Methods with Policy Convergence Guarantee (ICLR 2025 · similarity 0.90)
- A Temporal-Difference Approach to Policy Gradient Estimation (ICML 2022 · similarity 0.89)
- An Analytical Update Rule for General Policy Optimization (ICML 2022 · similarity 0.89)
- Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees (ICLR 2024 · similarity 0.88)
- Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.