Research map / Reinforcement learning / Policy gradient and actor-critic
What Makes Value Learning Efficient in Residual Reinforcement Learning?
Guozheng Ma, Lu Li, Haoyu Wang, Zixuan Liu, Pierre-Luc Bacon, Dacheng Tao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add base-policy warmup transitions to the replay buffer (ablation) | minimal residual SAC · ManiSkill (3 tasks) | sample efficiency / success: fails entirely without warmup on PegInsertionSide; improves consistently with more warmup | Improved |
| add explicit critic pre-training (Soft Q auto alpha, fixed alpha, Hard Q) on top of implicit warmup (ablation) | DAWN-style residual SAC · ManiSkill (3 tasks) | sample efficiency | Hurt |
| add exploration noise during warmup data collection instead of base policy only (ablation) | residual SAC warmup · ManiSkill | sample efficiency | No clear effect |
| add Layer Normalization to the critic (ablation) | residual SAC · ManiSkill (3 tasks) | sample efficiency | Improved |
| add critic normalization: critic sensitivity to residual action (ablation) | residual SAC · PegInsertionSide | gradient norm of Q wrt residual: approximately twice the level of the unnormalized baseline | Improved |
| use Layer Norm instead of Hyperspherical Norm in critic (ablation) | residual SAC · ManiSkill, varying lambda | success rate | Improved |
| replace MSE critic with distributional objectives (C51, QR, TQC) (ablation) | residual SAC with warmup and LN · ManiSkill (2 challenging tasks) | early-stage success | No clear effect |
| add normalization to the actor (ablation) | DAWN · ManiSkill/Adroit | success rate | No clear effect |
| remove either data-anchored warmup or critic normalization from DAWN (ablation) | DAWN · ManiSkill/Adroit | learning efficiency | Improved |
| use DAWN instead of Policy Decorator | Diffusion Policy residual RL · PegInsertionSide | time to 90% success: approximately 6x faster | Improved |
Closest papers by meaning
- Pretrain Value, Not Reward: Decoupled Value Policy Optimization (ICLR 2026 · similarity 0.90)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.89)
- Horizon Adaptive Offline Policy Learning via Value Stitching (NeurIPS 2026 · similarity 0.88)
- Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities (NeurIPS 2026 · similarity 0.87)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
