Research map / Reinforcement learning / Policy gradient and actor-critic
Ordering-based Conditions for Global Convergence of Policy Gradient Methods
Jincheng Mei, Bo Dai, Alekh Agarwal, Mohammad Ghavamzadeh, Csaba Szepesvari, Dale Schuurmans
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Swap columns 2 and 3 of the feature matrix (Example 1 to Example 2), which breaks reward order preservation (ablation) | Softmax PG · 4-action bandit, log-linear policy | limit of expected reward: 9 -> 8 | Hurt |
| Swap columns 1 and 2 of the feature matrix (Example 1 to Example 3), which breaks optimal action preservation (ablation) | NPG · 4-action bandit, log-linear policy | limit of expected reward: 9 -> 8 | Hurt |
| Swap columns 1 and 2 of the feature matrix (Example 1 to Example 3) (ablation) | Softmax PG · 4-action bandit, log-linear policy | limit of expected reward: 9 -> 9 | No clear effect |
| Swap columns 2 and 3 of the feature matrix (Example 1 to Example 2) (ablation) | NPG · 4-action bandit, log-linear policy | limit of expected reward: 9 -> 9 | No clear effect |
| Use a feature matrix with non-zero approximation error (14.2338) instead of requiring zero approximation error | Softmax PG and NPG · Example 1 | global convergence: 9 | No clear effect |
| Violate non-domination condition on features (Proposition 2) even with linearly realizable reward (ablation) | Softmax PG · K=3, d=2 counterexample | global convergence | Hurt |
| Make r-hat(a*) equal to r-hat(2) so strict inequality of optimal action preservation fails (ablation) | NPG · Example 4 | global convergence | Hurt |
| Remove reward order preservation on Example 5 (no order-preserving w exists) (ablation) | Softmax PG · Example 5 | limit of expected reward: 9 | No clear effect |
| Remove reward order preservation on Example 5 (ablation) | NPG · Example 5 | global convergence | Hurt |
| Run NPG with optimal action preservation satisfied | NPG · Example 1 | convergence rate | Improved |
| Run Softmax PG for 10^8 iterations | Softmax PG · Example 1 | convergence rate: slope approaches -1 | Mixed |
Closest papers by meaning
- Convergence Analysis of Policy Gradient Methods with Dynamic Stochasticity (ICML 2025 · similarity 0.88)
- Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates (NeurIPS 2024 · similarity 0.87)
- On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Achieving $\tilde{\mathcal{O}}(1/N)$ Optimality Gap in Restless Bandits through Gaussian Approximation (NeurIPS 2025 · similarity 0.86)
- Generalized Linear Bandits with Limited Adaptivity (NeurIPS 2024 · similarity 0.86)
- Learning Optimal Deterministic Policies with Stochastic Policy Gradients (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
