Research map / Reinforcement learning / Policy gradient and actor-critic
Optimal Estimation of Policy Gradient via Double Fitted Iteration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Estimate Q and its policy gradient jointly by iterative fitted Bellman regression instead of using importance sampling | Off-policy policy-gradient estimation · FrozenLake and CliffWalking | gradient estimation error | Helped |
| Use iterative fitted estimation rather than importance sampling as distribution mismatch grows | Off-policy policy-gradient estimation · FrozenLake and CliffWalking | gradient estimation error | Helped |
| Reuse data from the recent 5 iterations for gradient estimation | REINFORCE · On-policy policy optimization | convergence speed | Helped |
| Replace online gradients with FPG estimates from a fixed offline dataset | REINFORCE · Offline policy optimization | policy value | Helped |
| Use importance-sampling estimates from the same small fixed offline dataset | REINFORCE · Offline policy optimization | policy value | Hurt |
| Use a ReLU policy network instead of softmax tabular parameterization | FPG · CliffWalking | gradient estimation error | Helped |
| Increase the dataset size | FPG and importance sampling · CliffWalking, deep policy network | gradient estimation error | Helped |
| Increase distribution mismatch | FPG and importance sampling · FrozenLake | gradient estimation error | Hurt |
| Increase distribution mismatch | FPG and importance sampling · CliffWalking, deep policy network | gradient estimation error | Mixed |
| Bootstrap FPG estimates instead of IS estimates | Off-policy gradient confidence regions · Bootstrap inference | confidence-region size and accuracy | Helped |
| Increase bootstrap sample size | FPG bootstrap · Bootstrap inference | confidence-region concentration: K=200,500 | Helped |
Closest papers by meaning
- From Importance Sampling to Doubly Robust Policy Gradient (ICML 2020 · similarity 0.90)
- Deep Bayesian Quadrature Policy Optimization (AAAI 2021 · similarity 0.89)
- Statistically Efficient Off-Policy Policy Gradients (ICML 2020 · similarity 0.88)
- A Temporal-Difference Approach to Policy Gradient Estimation (ICML 2022 · similarity 0.88)
- PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimation (ICML 2022 · similarity 0.87)
- An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods (NeurIPS 2020 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.