Research map / Reinforcement learning / Policy gradient and actor-critic

Simple Policy Optimization

ICML 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Tree search and policy optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Simple Policy Optimization (SPO), an unconstrained first-order policy gradient objective that replaces PPO's ratio clipping with a quadratic penalty on the probability ratio, and shows it outperforms PPO on Atari and MuJoCo, especially with deeper networks.

Problem

PPO's ratio clipping fails to keep probability ratios within the trust region, leading to uncontrolled ratio deviations and collapse when training deeper policy networks, while TRPO's second-order constrained optimization is inefficient.

Approach

Replace the clipped surrogate with an unconstrained objective r*A - (|A|/(2*eps))*(r-1)^2, whose gradient never vanishes and points every sample's ratio toward the boundary 1+sign(A)*eps, motivated by a TV-divergence-based performance lower bound and an epsilon-aligned objective class.

Key results

• On MuJoCo with 3-layer policies, PPO outperforms SPO on Ant-v4 (5323.2 vs 4911.3) and HalfCheetah-v4 (4550.2 vs 3602.4), but SPO wins on Humanoid-v4 (2870.0 vs 795.1) and HumanoidStandup-v4 (152378.7 vs 143908.8).
• With 7-layer policies, PPO collapses on Ant-v4 (1002.8) while SPO reaches 4672.5, and on HalfCheetah-v4 SPO gets 5307.3 vs PPO 2242.3, with PPO's maximum ratio deviations reaching values in the hundreds to thousands while SPO stays around 0.19.
• SPO trained with a ResNet-18 encoder on Atari 2600 improves with capacity while keeping ratio deviation constrained, whereas PPO struggles to train large networks even with epsilon=0.1.

Measured effects reported in the paper

Surprising findings

• PPO's maximum average ratio deviation reaches values in the thousands on some deep-network MuJoCo runs (e.g., 3689.957 on Humanoid-v4 with 3 layers), far beyond the nominal clipping range.
• Under the 3-layer setting, PPO outperforms SPO on Ant-v4 and HalfCheetah-v4 despite SPO's tighter ratio control.

Limitations

• Performance is mixed on some MuJoCo tasks at standard network depth, where PPO sometimes outperforms SPO.
• Theoretical argument relies on the TV-divergence lower bound and an approximation linking clipping to TV divergence rather than a rigorous guarantee for the deployed objective.
• Evaluation is limited to Atari 2600 and MuJoCo-v4 with no language-model or robotics experiments.

Experimental setup

Datasets: Atari 2600 MuJoCo-v4

Benchmarks: Atari 2600 MuJoCo-v4

Metrics: average return IQM median mean optimality gap ratio deviation PPO-normalized score

Baselines: PPO-Clip PPO-Penalty SPU PPO-RB TR-PPO TR-PPO-RB RPO

Builds on: TRPO PPO Generalized Advantage Estimation

Closest papers by meaning

• Reflective Policy Optimization (ICML 2024 · similarity 0.89)
• Absolute Policy Optimization: Enhancing Lower Probability Bound of Performance with High Confidence (ICML 2024 · similarity 0.88)
• Embedding Safety into RL: A New Take on Trust Region Methods (ICML 2025 · similarity 0.88)
• ACPO: A Policy Optimization Algorithm for Average MDPs with Constraints (ICML 2024 · similarity 0.87)
• PPO in the Fisher-Rao geometry (NeurIPS 2026 · similarity 0.87)
• DPO Meets PPO: Reinforced Token Optimization for RLHF (ICML 2025 · similarity 0.87)