Research map / Optimization / Nonconvex and smooth optimization

$\phi$-Update: A Class of Policy Update Methods with Policy Convergence Guarantee

ICLR 2025 · Optimization › Nonconvex and smooth optimization · approach: Coordinate and Adam convergence analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a general policy update rule, phi-update, that scales advantage functions by a function phi, and establishes its global asymptotic state-value convergence under mild conditions on phi.

Problem

Policy update methods in reinforcement learning lack a unified framework with convergence guarantees; softmax natural policy gradient and Hadamard policy gradient share a similar update pattern that has not been generalized.

Approach

Generalizes the shared update pattern of softmax natural policy gradient and Hadamard policy gradient into a family of phi-updates, where phi is a scaling function applied to advantage functions, and proves global asymptotic state-value convergence under mild conditions on phi.

Key results

• Establishes global asymptotic state value convergence of phi-update under very mild conditions on phi (from abstract).

Experimental setup

Builds on: softmax natural policy gradient Hadamard policy gradient

Closest papers by meaning

• A Parametric Class of Approximate Gradient Updates for Policy Optimization (ICML 2022 · similarity 0.91)
• On the Global Convergence Rates of Softmax Policy Gradient Methods (ICML 2020 · similarity 0.90)
• Dr Jekyll & Mr Hyde: the strange case of off-policy policy updates (NeurIPS 2021 · similarity 0.90)
• Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework (NeurIPS 2026 · similarity 0.88)
• Improving Value Estimation Critically Enhances Vanilla Policy Gradient (ICML 2025 · similarity 0.87)
• Convergence Analysis of Policy Gradient Methods with Dynamic Stochasticity (ICML 2025 · similarity 0.86)