Research map / Reinforcement learning / Policy gradient and actor-critic

Mean Field Langevin Actor-Critic: Faster Convergence and Global Optimality beyond Lazy Learning

ICML 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Actor-critic theory

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes mean-field Langevin actor-critic (MFLTD critic, MFLPG actor) with two-layer neural networks in the mean-field regime, proving sublinear critic convergence and linear actor convergence to the KL-regularized global optimum with feature learning.

Problem

Theory of neural actor-critic methods: existing convergence guarantees rely on lazy/NTK training that does not capture feature learning, and TD learning with neural networks suffers from semi-gradient bias.

Approach

Treat the policy and Q-function as two-layer mean-field neural networks whose parameter distributions are updated by Langevin dynamics: policy gradient in Wasserstein space for the actor (MFLPG) and a proximal, double-loop TD update for the critic (MFLTD), yielding global optimality beyond the lazy regime.

Key results

• MFLTD critic converges to the true value function at a sublinear O(1/T_TD) rate with O(lambda_TD) regularization bias (Corollary 1).
• MFLPG actor achieves linear convergence in continuous time, max_pi J^pi - J[rho_t] <= 2R exp(-alpha lambda t) + O(lambda^{1/2}) (Theorem 2).
• Discretized particle analysis gives max_pi J^pi - E[J_{Theta_T}] = O(m^{-2}) for suitable parameter settings, improving on the O(m^{-1/16}) NTK rate of Wang et al. (2020).
• On CartPole-v1, mean-field training outperforms NTK-TD at equal neuron count and double-loop MFLTD reaches lower mean squared Bellman error than single-loop TD(1) at high accuracy.

Measured effects reported in the paper

Surprising findings

• Double-loop MFLTD is faster than single-loop TD(1) only in low-accuracy regions; at higher accuracy the single-loop method slows down.

Limitations

• Analysis relies on strong regularity assumptions (smooth bounded activations, Barron-type value function class, moment conditions on Radon-Nikodym derivatives).
• Numerical validation is limited to a single small environment (CartPole-v1) with 10 runs.
• Double-loop structure adds computational cost compared with single-loop TD.

Experimental setup

Datasets: CartPole-v1

Benchmarks: CartPole-v1

Metrics: mean squared Bellman error convergence rate bounds

Baselines: TD(1) NTK-TD Leahy et al. (2022) Wang et al. (2020) NTK policy gradient

Builds on: Mean-field Langevin dynamics (Nitanda et al., 2022; Chizat, 2022) NTK-based actor-critic analysis (Wang et al., 2020; Cai et al., 2019) Leahy et al. (2022) entropy-regularized policy gradient in Wasserstein gradient TD(0) semi-gradient

Closest papers by meaning

• Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning (ICLR 2025 · similarity 0.89)
• Finite-Time Global Optimality Convergence in Deep Neural Actor-Critic Methods for Decentralized Multi-Agent Reinforcement Learning (ICML 2025 · similarity 0.88)
• ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages (ICML 2024 · similarity 0.88)
• Closing the Gap: Achieving Global Convergence (Last Iterate) of Actor-Critic under Markovian Sampling with Neural Network Parametrization (ICML 2024 · similarity 0.88)
• Non-Asymptotic Analysis for Single-Loop (Natural) Actor-Critic with Compatible Function Approximation (ICML 2024 · similarity 0.87)
• Wasserstein Policy Optimization (ICML 2025 · similarity 0.87)