Research map / Optimization / Nonconvex and smooth optimization
Understanding the Effect of Stochasticity in Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace exact gradients with on-policy stochastic estimates for NPG | NPG · One-state finite-action MDP | global convergence: O(e^{-c · t}) -> fails w.p. >0 | Hurt |
| Replace exact gradients with on-policy stochastic estimates for GNPG | GNPG · One-state finite-action MDP | global convergence: O(e^{-c · t}) -> fails w.p. >0 | Hurt |
| Use on-policy stochastic gradients instead of exact gradients | Softmax PG · One-state finite-action MDP | convergence: Θ(1/t) -> O(1/√t) | Hurt |
| Use NPG rather than Softmax PG | One-state finite-action MDP, exact gradients · One-state finite-action MDP, exact gradients | sub-optimality convergence rate: Θ(1/t) -> O(e^{-c · t}) | Helped |
| Use GNPG rather than Softmax PG | One-state finite-action MDP, exact gradients · One-state finite-action MDP, exact gradients | sub-optimality convergence rate: Θ(1/t) -> O(e^{-c · t}) | Helped |
| Use Softmax PG rather than NPG | One-state finite-action MDP, on-policy stochastic gradients · One-state finite-action MDP, on-policy stochastic gradients | global convergence: fails w.p. >0 -> converges in prob. | Helped |
| Use Softmax PG rather than GNPG | One-state finite-action MDP, on-policy stochastic gradients · One-state finite-action MDP, on-policy stochastic gradients | global convergence: fails w.p. >0 -> converges in prob. | Helped |
| Use an oracle baseline informed by reward-gap information | NPG · One-state finite-action MDP, on-policy stochastic gradients | global convergence probability: positive failure probability -> probability 1 | Helped |
| Use an oracle baseline informed by reward-gap information | NPG · One-state finite-action MDP, on-policy stochastic gradients | sub-optimality convergence rate: O(e^{-c · t}) | Helped |
| Run independent NPG trials in parallel and select the best | NPG · One-state finite-action MDP | global convergence probability: With probability 1 - δ | Helped |
| Run independent NPG trials in parallel and select the best | NPG · One-state finite-action MDP | sub-optimality convergence rate: O(e^{-c · t}) | Helped |
Closest papers by meaning
- The Role of Baselines in Policy Gradient Optimization (NeurIPS 2022 · similarity 0.91)
- Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policies (ICML 2023 · similarity 0.88)
- Convergence Analysis of Policy Gradient Methods with Dynamic Stochasticity (ICML 2025 · similarity 0.87)
- Mollification Effects of Policy Gradient Methods (ICML 2024 · similarity 0.86)
- Learning Optimal Deterministic Policies with Stochastic Policy Gradients (ICML 2024 · similarity 0.86)
- Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.