Research map / Reinforcement learning / Policy gradient and actor-critic
Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an action-dependent rather than state-dependent baseline | Policy gradient · m-DimTargetMatching, 12 dimensions | solve time (iterations): 45.6 -> 45.6 | No clear effect |
| Use an action-dependent rather than state-dependent baseline | Policy gradient · m-DimTargetMatching, 100 dimensions | solve time (iterations): 150 -> 136; 9.3% speed improvement | Helped |
| Use an action-dependent rather than state-dependent baseline | Policy gradient · m-DimTargetMatching, 400 dimensions | solve time (iterations): 304 -> 268.2; 11.8% speed improvement | Helped |
| Use an action-dependent rather than state-dependent baseline | Policy gradient · m-DimTargetMatching, 2000 dimensions | solve time (iterations): 671.5 -> 595.5; 11.3% speed improvement | Helped |
| Use action-dependent baselines instead of state-only baselines | Policy gradient on Hopper, HalfCheetah, Ant, and hand manipulation · Continuous control benchmarks | learning performance | Helped |
| Estimate the marginalized Q baseline using the mean action instead of Monte Carlo samples (ablation) | Action-dependent baseline · Continuous control benchmarks | learning performance | No clear effect |
| Add access to the hidden hole location in the baseline | Policy gradient on blind peg insertion · Blind peg insertion | learning speed | Helped |
| Add the other agent's state and action to the shared baseline | Two-agent policy gradient · CommunicateTarget | training performance | Helped |
| Use an intermediate GAE setting (λ=0.97) rather than the high-bias or high-variance choices (ablation) | Policy gradient with action-dependent baselines · GAE study | learning performance: λ=0.97 | Helped |
Closest papers by meaning
- The Mirage of Action-Dependent Baselines in Reinforcement Learning (ICML 2018 · similarity 0.95)
- Variance Reduction for Reinforcement Learning in Input-Driven Environments (ICLR 2019 · similarity 0.91)
- Action-Level Behavior Policy Optimization for Variance-Reduced Policy Gradients (NeurIPS 2026 · similarity 0.90)
- Action-dependent Control Variates for Policy Optimization via Stein Identity (ICLR 2018 · similarity 0.90)
- On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Method (NeurIPS 2021 · similarity 0.90)
- Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Buffers (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.