Research map / Reinforcement learning / Policy gradient and actor-critic
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use conservative Q-Prop (disable control variate when covariance estimate is negative) instead of standard/aggressive Q-Prop (ablation) | Q-Prop on TRPO · HalfCheetah-v1 | stability of average return | Helped |
| Apply Q-Prop control variate (conservative variant) with TRPO update instead of TRPO alone | TRPO · HalfCheetah-v1 | sample efficiency (episodes to reward 4000): about 10x fewer samples | Helped |
| Reduce batch size with conservative Q-Prop instead of TRPO's large batches (ablation) | Q-Prop vs TRPO · HalfCheetah-v1 | sample efficiency across batch sizes | Helped |
| Use vanilla policy gradient with Q-Prop control variate instead of plain VPG (ablation) | Vanilla policy gradient · HalfCheetah-v1 | average return | Helped |
| Apply Q-Prop (TR-c-Q-Prop) instead of TRPO on Humanoid | TRPO · Humanoid-v1 | max return and convergence episodes | Helped |
| Use Q-Prop instead of DDPG on Humanoid | DDPG · Humanoid-v1 | max return | Helped |
| Use Q-Prop instead of TRPO on Ant, Hopper, Walker | TRPO · Ant / Hopper / Walker | max return | Mixed |
| Use Q-Prop instead of DDPG on Hopper and Walker | DDPG · Hopper / Walker | max return | Mixed |
| Reward scaling tuned for DDPG (0.1) versus default (ablation) | DDPG · HalfCheetah-v1 | average return | Helped |
| Use the off-policy critic for control variate in Q-Prop instead of DDPG's unstable off-policy learning | Q-Prop · HalfCheetah-v1 | stability across hyperparameters | Helped |
Closest papers by meaning
- Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (NeurIPS 2017 · similarity 0.90)
- Diffusion Policies Creating a Trust Region for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Simplifying Deep Temporal Difference Learning (ICLR 2025 · similarity 0.89)
- Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.89)
- Actor-Free Continuous Control via Structurally Maximizable Q-Functions (NeurIPS 2025 · similarity 0.89)
- Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.