Research map / Reinforcement learning / Policy gradient and actor-critic

Mollification Effects of Policy Gradient Methods

ICML 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Trajectory reuse in policy gradients · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that Gaussian-policy gradient methods implicitly solve a backward heat equation (mollifying the objective), explains why they work on fractal landscapes and why small or large policy variance hurts, and illustrates this on hopper, double pendulum and planar quadrotor control tasks.

Problem

Policy gradient methods are applied to continuous control where the RL objective can be non-smooth or fractal (chaotic dynamics), so the true gradient may not exist; it is unclear why they still work and what the role of policy variance is.

Approach

Views the Gaussian policy's density as a smoothing kernel that mollifies the objective; shows that the mean-variance objective L_s(mu, sigma^2) solves a heat equation, so stochastic policy gradient ascent is gradient ascent on a heat-equation solution. Deterministic-policy convergence then corresponds to a backward (ill-posed) heat problem, and the uncertainty principle gives a smoothness-vs-fidel

Key results

• On hopper stand, sigma=0.005 gives the highest return variance and sigma=0.05 and 0.5 train stably, consistent with an optimal intermediate variance.
• Double pendulum: fine-tuning an initial LQR-like linear policy with policy gradient stabilizes the system after 50 epochs.
• Planar quadrotor: even starting from the optimal policy, the training curve immediately drops after one epoch because the optimum is a narrow spike filtered by the Gaussian kernel.

Measured effects reported in the paper

Surprising findings

• Starting exactly at the optimal quadrotor policy, policy gradient immediately degrades and never returns to it.
• Smaller policy variance can increase training variance rather than reduce it, due to weaker mollification on fractal landscapes.

Limitations

• Theory assumes the value function is exact and focuses only on the policy improvement stage.
• Experiments are small-scale simulated control tasks with linear or 2-layer policies, with few seeds.
• Results on Gaussian-kernel mollification do not cover non-Gaussian or non-isotropic policies beyond a diagonal-covariance extension.

Experimental setup

Datasets: OpenAI Gym Hopper (simulated) double pendulum simulation planar quadrotor simulation

Benchmarks: Hopper stand double pendulum stabilization planar quadrotor balance

Metrics: total reward per epoch variance of return sequence

Builds on: REINFORCE policy gradient theorem Gaussian smoothing

Closest papers by meaning

• Fractal Landscapes in Policy Optimization (NeurIPS 2023 · similarity 0.89)
• Learning Optimal Deterministic Policies with Stochastic Policy Gradients (ICML 2024 · similarity 0.88)
• Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
• Occupancy-based Policy Gradient: Estimation, Convergence, and Optimality (NeurIPS 2024 · similarity 0.87)
• Convergence Analysis of Policy Gradient Methods with Dynamic Stochasticity (ICML 2025 · similarity 0.87)
• The Ladder in Chaos: Improving Policy Learning by Harnessing the Parameter Evolving Path in A Low-dimensional Space (NeurIPS 2024 · similarity 0.86)