Research map / Reinforcement learning / Policy gradient and actor-critic
Batch Reinforcement Learning With a Nonparametric Off-Policy Policy Gradient
Samuele Tosatto, Joao Carvalho, Jan Peters
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NOPG nonparametric gradient instead of importance-sampling G(PO)MDP on off-policy LQG data | gradient estimator · LQG gradient analysis | bias and variance of gradient estimate: significantly both lower bias and variance | Improved |
| increase off-policiness alpha of the dataset (ablation) | importance-sampling G(PO)MDP · LQG gradient analysis | variance | Hurt |
| increase off-policiness alpha of the dataset (ablation) | NOPG · LQG gradient analysis | bias | Hurt |
| decrease kernel bandwidth (ablation) | NOPG-S · LQG gradient analysis (1000 samples, alpha 0.5) | variance | Hurt |
| increase kernel bandwidth (ablation) | NOPG-S · LQG gradient analysis (1000 samples, alpha 0.5) | bias | Hurt |
| policy moves toward low-density regions (nonparametric gradient) | NOPG · Pendulum-v0 trust region analysis | gradient magnitude | Hurt |
| use NOPG instead of DDPG, TD3, SAC | policy optimization · Pendulum-v0 and Cart-Pole | sample efficiency: more than one order of magnitude | Improved |
| use NOPG instead of BEAR, BRAC, MOPO, MOReL | offline policy optimization · Pendulum-v0, Cart-Pole, U-Maze | sample efficiency | Mixed |
| use NOPG instead of behavioral cloning on human demonstrations | offline learning · Mountain Car (human demonstrations) | return: optimal policy with two demonstrated trajectories | Improved |
| apply NOPG to higher-dimensional task | NOPG · Hopper (D4RL) | return | No clear effect |
Closest papers by meaning
- Statistically Efficient Off-Policy Policy Gradients (ICML 2020 · similarity 0.91)
- Combining parametric and nonparametric models for off-policy evaluation (ICML 2019 · similarity 0.91)
- Provably Good Batch Off-Policy Reinforcement Learning Without Great Exploration (NeurIPS 2020 · similarity 0.91)
- Continuous Doubly Constrained Batch Reinforcement Learning (NeurIPS 2021 · similarity 0.90)
- On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples (AAAI 2023 · similarity 0.90)
- Critic Regularized Regression (NeurIPS 2020 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
