Research map / Reinforcement learning / Policy gradient and actor-critic
Variance Penalized On-Policy and Off-Policy Actor-Critic
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add direct return-variance estimation and penalize variance in the PPO objective | PPO · HalfCheetah | Var (1e5): 1.6 -> 0.1 (93%) | Helped |
| Add direct return-variance estimation and penalize variance in the PPO objective | PPO · Hopper | Var (1e5): 6.6 -> 4.0 (39.4%) | Helped |
| Add direct return-variance estimation and penalize variance in the PPO objective | PPO · Walker2d | Var (1e5): 12.1 -> 9.2 (23.9%) | Helped |
| Add direct return-variance estimation and penalize variance in the PPO objective | PPO · HalfCheetah | Mean: 1557 -> 1373 | Hurt |
| Add direct return-variance estimation and penalize variance in the PPO objective | PPO · Hopper | Mean: 1944 -> 1624 | Hurt |
| Add direct return-variance estimation and penalize variance in the PPO objective | PPO · Walker2d | Mean: 3058 -> 2625 | Hurt |
| Increase the mean-variance trade-off parameter ψ | VPAC · Continuous puddle-world | Variance in return | Helped |
| Increase the mean-variance trade-off parameter ψ | VPAC · Continuous puddle-world | Mean return | Hurt |
| Use a very high mean-variance trade-off parameter ψ | VPAC · Four rooms | Mean return | Hurt |
Closest papers by meaning
- Policy Evaluation for Variance in Average Reward Reinforcement Learning (ICML 2024 · similarity 0.91)
- Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning (AAAI 2026 · similarity 0.90)
- Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines (ICLR 2018 · similarity 0.89)
- Variance-Aware Off-Policy Evaluation with Linear Function Approximation (NeurIPS 2021 · similarity 0.89)
- Statistically Efficient Off-Policy Policy Gradients (ICML 2020 · similarity 0.89)
- Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.