Research map / Optimization / Nonconvex and smooth optimization
The Role of Baselines in Policy Gradient Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a state value baseline to on-policy stochastic NPG (ablation) | NPG with on-policy importance sampling · One-state MDP with bounded stochastic rewards | global convergence to an optimal policy: positive probability of suboptimal convergence -> almost sure global convergence | Helped |
| Add a state value baseline to on-policy stochastic NPG | NPG with on-policy importance sampling · One-state MDP with bounded stochastic rewards | suboptimality gap: O(1/t) | Helped |
| Add a state value baseline to on-policy stochastic NPG | NPG with on-policy importance sampling · Finite MDPs with sufficiently exploratory initial state distribution | value suboptimality gap: O(1/t) | Helped |
| Add a state value baseline (ablation) | On-policy stochastic NPG · One-state MDP, fixed repeated-action sampling | update aggressiveness: O(e^{-c · t}) -> Ω(1/t) | Helped |
| Add a state value baseline (ablation) | On-policy stochastic NPG · One-state MDP with stochastic rewards | update variance: unbounded -> unbounded | No clear effect |
| Use a value baseline to moderate updates | On-policy stochastic NPG · One-state MDP, adversarial initialization | expected reward: r(2) ≈ 0.95 -> r(1) ≈0.97 | Helped |
| Use a value baseline | On-policy stochastic NPG · One-state MDP, uniform initialization, deterministic rewards | suboptimality-gap decay: slope is approximately -1; C′/ t | Helped |
Closest papers by meaning
- Understanding the Effect of Stochasticity in Policy Optimization (NeurIPS 2021 · similarity 0.91)
- Beyond Variance Reduction: Understanding the True Impact of Baselines on Policy Optimization (ICML 2021 · similarity 0.91)
- The Mirage of Action-Dependent Baselines in Reinforcement Learning (ICML 2018 · similarity 0.90)
- Convergence Analysis of Policy Gradient Methods with Dynamic Stochasticity (ICML 2025 · similarity 0.88)
- Mollification Effects of Policy Gradient Methods (ICML 2024 · similarity 0.87)
- Learning Optimal Deterministic Policies with Stochastic Policy Gradients (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.