Research map / Reinforcement learning / Policy gradient and actor-critic

Mirror descent actor-critic methods for entropy regularised MDPs in general spaces: stability and convergence

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Actor-critic theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proves stability and convergence of a single-loop and a multi-step TD actor-critic with policy mirror descent for entropy-regularised MDPs in Polish state/action spaces under linear Q-realisability.

Problem

Stability and convergence of actor-critic with policy mirror descent and TD critic for entropy-regularised MDPs in general (Polish) state and action spaces, where KL divergence to the reference measure may be unbounded.

Approach

Derives sufficient step-size conditions for a single-loop TD actor-critic to keep KL bounded and converge, then introduces a multi-step TD variant where the number of inner TD steps grows logarithmically (sublinear rate) or linearly (linear rate under concentrability) with policy updates.

Key results

• Single-step TD actor-critic is stable (bounded KL and critic parameters) under step-size conditions, with linear convergence of the single-loop variant under restrictive conditions.
• With M(n) >= (4/(h Gamma)) log(c(n+1)) inner TD steps, the iterates are stable and the value gap satisfies min_r (V^{pi^r} - V*) <= a/n.
• Under a concentrability assumption and M(n) linear in n, the value gap decays as b exp(-min{1/xi, c} n).

Surprising findings

• A single TD step per policy update may be insufficient for stability in Polish action spaces for all discount factors, motivating multi-step TD.

Limitations

• Analysis is in the population (exact integrals) setting; no sample-based TD guarantees.
• Requires Q-realisability with linear features and restrictive step-size conditions for the single-step variant.
• Sublinear/linear rates are in number of policy updates; total TD steps scale as O(n log n) or O(n^2).

Experimental setup

Builds on: policy mirror descent TD learning TRPO PPO

Closest papers by meaning

• Sharp Convergence and Sample Complexity of Policy Mirror Descent for Average-Reward MDPs (NeurIPS 2026 · similarity 0.90)
• Convergence of Policy Mirror Descent Beyond Compatible Function Approximation (ICML 2025 · similarity 0.89)
• Policy Mirror Descent with Lookahead (NeurIPS 2024 · similarity 0.87)
• Operator World Models for Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
• On the Convergence of Single-Timescale Actor-Critic (NeurIPS 2025 · similarity 0.87)
• Learning mirror maps in policy mirror descent (ICLR 2025 · similarity 0.87)