Research map / Reinforcement learning / Policy gradient and actor-critic

PPO in the Fisher-Rao geometry

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Fisher-Rao PPO (FR-PPO), replacing PPO's clipped TV-based surrogate with a squared Fisher-Rao (FR²) penalty on squared densities, yielding monotonic policy improvement and sub-linear convergence guarantees with light empirical validation on Atari and MuJoCo.

Problem

PPO's clipped surrogate lacks formal guarantees for policy improvement and convergence because the implicit TV penalty is non-smooth and not a Bregman divergence.

Approach

Derive a tighter lower bound on the value difference with a TV² term weighted by occupancy measures, bound it by the squared Fisher-Rao distance between squared densities (a Bregman divergence generated by chi-squared), and use this as a mirror-descent-style penalty that yields a drop-in PPO loss.

Key results

• On MuJoCo, FR-PPO normalized reward 4.4 vs PPO 5.4, but FR-PPO wins more environments (3 vs 2 by max-sum metric).
• On Atari, FR-PPO normalized reward 5.6 vs PPO 5.8, with both winning 3 environments.
• Theory: monotonic policy improvement for FR-PPO and sub-linear O(1/n) last-iterate convergence in direct parametrization.

Surprising findings

• Empirically, FR-PPO does not clearly beat PPO; its normalized reward is lower on MuJoCo (4.4 vs 5.4) and Atari (5.6 vs 5.8).

Limitations

• Empirical evaluation is limited and focused on theory; FR-PPO does not clearly outperform PPO.
• Analysis requires assumptions such as relative concentrability and compact action spaces with semicontinuity.
• The penalty parameter tau seems to matter more than the PPO clipping parameter, requiring tuning.

Experimental setup

Datasets: Atari (BeamRider, Breakout, KungFuMaster, MsPacman, Pong, SpaceInvaders) MuJoCo (Hopper, HalfCheetah, Reacher, Swimmer, Walker)

Benchmarks: Atari MuJoCo

Metrics: normalized reward number of environments won

Baselines: PPO

Builds on: PPO TRPO Policy mirror descent Simple Policy Optimization

Closest papers by meaning

• On Stationary Point Convergence of PPO-Clip (ICLR 2024 · similarity 0.91)
• PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping (AAAI 2024 · similarity 0.88)
• Simple Policy Optimization (ICML 2025 · similarity 0.87)
• DPO Meets PPO: Reinforced Token Optimization for RLHF (ICML 2025 · similarity 0.86)
• Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.86)
• Solving Minimum-Cost Reach Avoid using Reinforcement Learning (NeurIPS 2024 · similarity 0.86)