Research map / Reinforcement learning / Policy gradient and actor-critic
Characterizing the Gap Between Actor-Critic and Policy Gradient
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a residual critic and its correction to the actor update (Res-AC) | Actor-critic · FourRoom | sample efficiency | Helped |
| Add a residual critic and its correction to the actor update (Res-AC) | Actor-critic · FourRoom | final return | Helped |
| Use the Actor_g update instead of the Res-AC update | Actor-critic · FourRoom | environment steps to similar final return: over 150,000 additional environment steps | Hurt |
| Add a residual-critic correction to SAC (Res-SAC) | SAC · Pendulum-v0, Reacher-v2, HalfCheetah-v2 | sample efficiency | Helped |
| Add a residual-critic correction to SAC (Res-SAC) | SAC · Reacher-v2 | asymptotic performance | Helped |
| Use the Stackelberg actor update instead of the standard SAC actor update | SAC · Pendulum-v0, Reacher-v2, HalfCheetah-v2 | performance | No clear effect |
| Perform 10 critic updates per actor update instead of one (ablation) | SAC · Pendulum-v0, Reacher-v2, HalfCheetah-v2 | performance | Helped |
| Use temporal-difference critic updates instead of Bellman-residual minimization (ablation) | Actor_o-Critic and Actor_g-Critic · FourRoom dynamic programming | performance | Helped |
| Use Bellman-residual critic updates instead of temporal-difference critic updates (ablation) | Policy Gradient · FourRoom dynamic programming | policy performance | No clear effect |
Closest papers by meaning
- Stackelberg Actor-Critic: Game-Theoretic Reinforcement Learning Algorithms (AAAI 2022 · similarity 0.92)
- Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation (ICML 2024 · similarity 0.89)
- Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning (ICML 2024 · similarity 0.89)
- Actor-Critics Can Achieve Optimal Sample Efficiency (ICML 2025 · similarity 0.89)
- On the Second-Order Convergence of Biased Policy Gradient Algorithms (ICML 2024 · similarity 0.88)
- Decoupled Policy Actor-Critic: Bridging Pessimism and Risk Awareness in Reinforcement Learning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.