Research map / Reinforcement learning / Policy gradient and actor-critic

Towards Convergence of PPO: An Approximate Descent Approach

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Trajectory reuse in policy gradients

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Provides a theoretical convergence analysis of PPO's actor update (clipped surrogate, multiple minibatch epochs, rollout reuse) via an approximate descent framework.

Problem

Existing convergence analyses of PPO do not fully capture its actor update structure: clipped surrogate gradients, multiple epochs of minibatch updates, and reuse of rollout data.

Approach

Analyzes PPO's actor update through an approximate descent lens so that the combined effects of clipping, multi-epoch minibatch updates and data reuse are covered by a convergence guarantee.

Experimental setup

Builds on: PPO

Closest papers by meaning

• PPO in the Fisher-Rao geometry (NeurIPS 2026 · similarity 0.91)
• TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing (NeurIPS 2026 · similarity 0.90)
• Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.89)
• ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages (ICML 2024 · similarity 0.89)
• PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping (AAAI 2024 · similarity 0.88)
• REBEL: Reinforcement Learning via Regressing Relative Rewards (NeurIPS 2024 · similarity 0.88)