Research map / Reinforcement learning / Policy gradient and actor-critic
Deep Conservative Policy Iteration
Nino Vieillard, Olivier Pietquin, Matthieu Geist
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CPI-style stochastic policy mixtures with α0=0.1 | DQN on Cartpole · Cartpole (500-step) | average episode score: 300 -> 480 | Improved |
| add CPI-style stochastic policy mixtures with α0=0.1 | DQN on Cartpole · Cartpole (500-step) | standard deviation of score: approximately 200 -> around 20 | Improved |
| use a constant small mixture rate (α0=0.001) (ablation) | DCPI on Cartpole · Cartpole (500-step) | stability | Improved |
| use a constant higher mixture rate (ablation) | DCPI on Cartpole · Cartpole (500-step) | stability | Hurt |
| replace a fixed mixture rate with an adaptive mixture rate (ablation) | DCPI on Cartpole · Cartpole (500-step) | convergence speed and stability | Improved |
| set the mixture rate to α=1, removing conservative mixing (ablation) | DCPI on Cartpole · Cartpole (500-step) | stability | Improved |
| use the SPI adaptive rate, clipped between 0 and 1 (ablation) | DCPI on Cartpole · Cartpole (500-step) | training behavior | No clear effect |
| use the Adamax adaptive mixture rate (ablation) | DCPI on Cartpole · Cartpole (500-step) | convergence speed and stability | Mixed |
| use the CPI adaptive rate with α0=0.1 (ablation) | DCPI on Cartpole · Cartpole (500-step) | average episode score: 300 -> 480 | Improved |
| act with an ε-greedy policy derived from qθ rather than πω (ablation) | DCPI on Atari · Atari | performance | Improved |
| add the DCPI conservative actor and adaptive mixture rate | DQN on Atari · Atari (20-game subset) | games outperforming DQN: 15 games over 20 | Improved |
| use a conservative mixture rate | DQN on Atari · Atari | learning stability | No clear effect |
Closest papers by meaning
- Projections for Approximate Policy Iteration Algorithms (ICML 2019 · similarity 0.87)
- Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework (NeurIPS 2026 · similarity 0.87)
- Multi-step Greedy Reinforcement Learning Algorithms (ICML 2020 · similarity 0.87)
- Dual Policy Iteration (NeurIPS 2018 · similarity 0.87)
- Accelerated Policy Gradient: On the Convergence Rates of the Nesterov Momentum for Reinforcement Learning (ICML 2024 · similarity 0.86)
- Convergence Analysis of Policy Gradient Methods with Dynamic Stochasticity (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
