Research map / Reinforcement learning / Policy gradient and actor-critic

Consensus Based Stochastic Optimal Control

ICML 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Variance-reduced policy gradients

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes gradient-free consensus-based optimization (M-CBO and Adam-CBO) to learn feedback policies for high-dimensional finite-horizon stochastic optimal control, with a convergence proof for M-CBO.

Problem

Solving high-dimensional, finite-horizon, model-free stochastic optimal control problems, where policy gradient estimates from Monte Carlo sampling have high variance and HJB/BSDE methods scale poorly or require transition-kernel modeling.

Approach

Replaces policy-gradient optimization with interacting-particle consensus dynamics (with momentum and an adaptive Adam-like timestep/second-moment scaling) where agents move toward a Gibbs-weighted consensus of policy parameters evaluated via Monte Carlo cost estimates, plus adjustable Gaussian exploration noise; proves mean-field convergence of M-CBO.

Key results

• On LQG control in 1-16 dimensions, Adam-CBO's value estimates are closer to MC reference than the BSDE method, especially for the nonconvex double-well terminal cost.
• On Pendulum-v1 and CartPole-v1, Adam-CBO converges to optimal value faster in learning efficiency, but takes longer wall-clock time (1124.88 vs 145-576 for baselines over 100k steps).
• Theory: M-CBO mean-field limit energy decays exponentially under suitable parameter choices (E[rho_t] <= E[rho_0] exp(-(1-tau) lambda t)).

Measured effects reported in the paper

Surprising findings

• Adam-CBO has the highest runtime among compared RL methods despite faster convergence in learning efficiency.
• Value estimates on LQG show accuracy sensitive to training batch size.

Limitations

• Convergence proven only for M-CBO, not Adam-CBO, under assumptions on the cost function.
• Accuracy sensitive to training batch size.
• Not a direct comparison with gradient-based RL methods; higher computational cost.

Experimental setup

Datasets: simulated controlled dynamics (LQG, Ginzburg-Landau, systemic risk mean-field, multi-agent robotic navigation) Pendulum-v1 CartPole-v1

Benchmarks: linear quadratic Gaussian control (1-16D) Ginzburg-Landau model systemic risk mean-field control multi-agent robotic navigation Pendulum-v1 CartPole-v1

Metrics: value function estimate error vs MC reference cost / value function convergence runtime

Baselines: BSDE method DDPG PPO SAC TD3 TQC CrossQ DQN

Builds on: Consensus-based optimization (Fornasier et al., 2024) Adam-CBO (Chen et al., 2022) deep BSDE method (Han et al., 2018)

Closest papers by meaning

• Deep learning for continuous-time stochastic control with jumps (NeurIPS 2025 · similarity 0.87)
• Learning Optimal Deterministic Policies with Stochastic Policy Gradients (ICML 2024 · similarity 0.87)
• A Differential and Pointwise Control Approach to Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
• Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.85)
• Convergence Analysis of Policy Gradient Methods with Dynamic Stochasticity (ICML 2025 · similarity 0.85)
• Mean Field Langevin Actor-Critic: Faster Convergence and Global Optimality beyond Lazy Learning (ICML 2024 · similarity 0.85)