Research map / Reinforcement learning / Policy gradient and actor-critic
A Theoretical Justification for Asymmetric Actor-Critic Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a finite-time expectation bound for asymmetric (privileged-critic) natural actor-critic with agent-state policies, showing the asymmetric critic removes the aliasing error term present in symmetric actor-critic.
Problem
Asymmetric actor-critic methods for partially observable RL are widely used and shown to be unbiased in theory, but there is no theoretical justification for their faster convergence compared with symmetric learning, especially under agent-state aliasing.
Approach
Adapts the finite-time natural actor-critic analysis to an asymmetric critic conditioned on (state, agent state, action) with linear function approximation, and compares it to the symmetric critic bound; the asymmetric bound has no aliasing term, so asymmetric learning is less sensitive to agent-state aliasing.
Key results
- Finite-time bound for asymmetric m-step TD: the aliasing term present in the symmetric bound is absent, leaving only TD, approximation and distribution-shift errors.
- For the natural actor-critic, the inference error term epsilon_inf is zero for the asymmetric algorithm, whereas it is nonzero for the symmetric one.
- The asymmetric gradient residual error epsilon_grad may be higher than the symmetric one, even in the tabular case, so the trade-off is not uniformly favorable.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an asymmetric critic conditioned on environment state and agent state instead of a symmetric critic conditioned only on agent state | Natural actor-critic with agent-state policies · Partially observable environments | critic approximation error bound | Helped |
| Use an asymmetric critic instead of a symmetric critic | Natural actor-critic with agent-state policies · Partially observable environments | policy suboptimality bound | Helped |
| Use the asymmetric natural policy-gradient loss rather than the symmetric loss | Natural actor-critic · Partially observable environments | natural policy-gradient approximation residual | Hurt |
Surprising findings
- The asymmetric residual error on the natural policy gradient may exceed the symmetric one even in the tabular case.
Limitations
- Analysis assumes a fixed agent-state process and linear function approximators; extension to learnable agent states and nonlinear approximators is left open.
- Bounds are given in expectation and do not address variance of the Q-function error.
- The asymmetric gradient residual error can be larger than the symmetric one.
Experimental setup
Metrics: finite-time error bound on Q-function approximation suboptimality bound of natural actor-critic policy
Baselines: symmetric natural actor-critic
Builds on: natural actor-critic finite-time analysis asymmetric actor-critic with history-state value function temporal difference learning with linear function approximation
Closest papers by meaning
- Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation (ICML 2024 · similarity 0.89)
- Provable Partially Observable Reinforcement Learning with Privileged Information (NeurIPS 2024 · similarity 0.89)
- Dual Critic Reinforcement Learning under Partial Observability (NeurIPS 2024 · similarity 0.88)
- Actor-Critics Can Achieve Optimal Sample Efficiency (ICML 2025 · similarity 0.88)
- Actor-Free Continuous Control via Structurally Maximizable Q-Functions (NeurIPS 2025 · similarity 0.88)
- ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.