Research map / Reinforcement learning / Policy gradient and actor-critic
Beyond Variance Reduction: Understanding the True Impact of Baselines on Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a more negative baseline instead of a positive perturbation of the minimum-variance baseline (ablation) | NPG with softmax on the three-arm bandit · Three-arm bandit, rewards (1, 0.7, 0) | convergence to the optimal policy | Hurt |
| Use the minimum-variance baseline instead of a positive perturbation (ablation) | NPG with softmax on the three-arm bandit · Three-arm bandit, rewards (1, 0.7, 0) | convergence to the optimal policy | Hurt |
| Use the value function baseline instead of the minimum-variance baseline (ablation) | NPG with softmax on the three-arm bandit · Three-arm bandit, rewards (1, 0.7, 0) | convergence to the optimal policy | Helped |
| Use a gap baseline between the optimal reward and the second-best reward rather than the minimum-variance baseline (ablation) | NPG with softmax · Three-arm bandit | probability of convergence to the optimal policy | Helped |
| Use a fixed negative baseline b<0 instead of a baseline in (0,1) (ablation) | NPG with sigmoid parameterization · Two-arm bandit | convergence to the optimal policy | Hurt |
| Increase the stepsize while using baseline b=-1 (ablation) | NPG · Two-arm bandit, 100 runs | suboptimal convergence rate: 5% -> 14% -> 22% | Hurt |
| Use off-policy sampling with importance sampling and sufficiently frequent action coverage instead of on-policy sampling (ablation) | NPG · Multi-armed bandit with bounded rewards and a unique optimal action | convergence to the optimal policy | Helped |
| Use larger rather than smaller baselines (ablation) | NPG in the four-rooms MDP · Four-rooms gridworld | discounted return | Mixed |
| Use a positive rather than negative perturbation of the minimum-variance baseline (ablation) | Vanilla PG with perturbations scaled to update magnitude · Four-rooms gridworld | discounted return | Hurt |
| Use a negative rather than positive constant baseline (ablation) | Vanilla PG / REINFORCE · Four-rooms gridworld | convergence to an optimal policy | Hurt |
| Use a negative rather than positive constant baseline (ablation) | Vanilla PG · 5x5 gridworld with two goal states | convergence to the optimal policy | Hurt |
Closest papers by meaning
- Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates (NeurIPS 2024 · similarity 0.91)
- Policy Optimization for CMDPs with Bandit Feedback: Learning Stochastic and Adversarial Constraints (ICML 2025 · similarity 0.89)
- Learning Optimal Deterministic Policies with Stochastic Policy Gradients (ICML 2024 · similarity 0.88)
- Structure Matters: Dynamic Policy Gradient (NeurIPS 2025 · similarity 0.88)
- Non-stationary Bandit Convex Optimization: A Comprehensive Study (NeurIPS 2025 · similarity 0.88)
- Selective Uncertainty Propagation in Offline RL (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.