Research map / Reinforcement learning / Policy gradient and actor-critic
The Mirage of Action-Dependent Baselines in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a learned state-action-dependent baseline instead of a learned state-dependent baseline | Policy gradient methods on continuous-control benchmarks · Humanoid and HalfCheetah | gradient estimator variance | No clear effect |
| Use an optimal state-action-dependent baseline instead of a state-dependent baseline | Policy gradient with discounted-return estimates · Humanoid and HalfCheetah | gradient estimator variance | No clear effect |
| Use an optimal state-action-dependent baseline instead of a state-dependent baseline | Policy gradient with GAE estimates · Humanoid and HalfCheetah | gradient estimator variance | Helped |
| Add a state-dependent baseline to the GAE estimator | Policy gradient with an imperfect value-function approximator · Continuous-control benchmarks | gradient estimator variance | Helped |
| Normalize only the learning signal, not the bias-correction term | Q-Prop · HalfCheetah and Humanoid | mean episode reward | Helped |
| Normalize all estimator terms to preserve an unbiased gradient (ablation) | Q-Prop · HalfCheetah and Humanoid | mean episode reward | No clear effect |
| Normalize only the learning signal, not the bias-correction term | Q-Prop · CartPole | mean episode reward | No clear effect |
| Fit the state-action-dependent baseline on the current batch before the policy update | Stein control variate · Humanoid | mean episode reward | Helped |
| Fit the state-action-dependent baseline on the current batch before the policy update | Stein control variate · HalfCheetah | training stability | Hurt |
| Use an extra policy action sample rather than importance-weighted batch samples for the bias correction (ablation) | Stein control variate · Continuous-control benchmarks | mean episode reward | Hurt |
| Fit a state-dependent baseline after the policy update instead of a state-action-dependent baseline (ablation) | Unbiased Stein control variate · Humanoid and HalfCheetah | mean episode reward | No clear effect |
| Use a horizon-aware value-function parameterization | TRPO · HalfCheetah, Walker2d, and Humanoid | mean episode reward | Helped |
Closest papers by meaning
- Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines (ICLR 2018 · similarity 0.95)
- Action-Level Behavior Policy Optimization for Variance-Reduced Policy Gradients (NeurIPS 2026 · similarity 0.90)
- The Role of Baselines in Policy Gradient Optimization (NeurIPS 2022 · similarity 0.90)
- Variance Reduction for Reinforcement Learning in Input-Driven Environments (ICLR 2019 · similarity 0.90)
- Action-dependent Control Variates for Policy Optimization via Stein Identity (ICLR 2018 · similarity 0.90)
- Efficient Multi-Policy Evaluation for Reinforcement Learning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.