Research map / Reinforcement learning / Policy gradient and actor-critic

Robust Reinforcement Learning with General Utility

NeurIPS 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Policy gradients for special settings · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes robust reinforcement learning with general (convex) utility as a minimax problem over policies and s-rectangular polyhedral transition ambiguity sets, with a two-phase stochastic GDA for stationarity and a vertex-enumeration algorithm with O(1/K) global convergence.

Problem

Standard RL with general utility assumes a fixed transition kernel, so policies may fail when deployed in an environment that differs from the simulator; robust RL has only been studied for linear utilities.

Approach

Formulates robust RL with general utility as minθ maxξ f(λθ,ξ) over occupancy measures, which subsumes robust RL, constrained RL, entropy-regularized and pure exploration variants. For convex f it gives a two-phase projected stochastic GDA with stationarity guarantees, and on s-rectangular polyhedral ambiguity sets it shows the inner maximizer lies at vertices, enabling a globally convergent algor

Key results

• Two-phase algorithm's projected gradient norm decays and converges in a small tabular simulation (10 states, 5 actions), matching Theorem 2.
• Global algorithm on s-rectangular polyhedral ambiguity sets achieves O(1/K) convergence of Γ(θK) to the robust optimum (Theorem 3).
• Gradient-convergence sample complexity is O(log^2[(1-γ)^-1 ε^-1] (1-γ)^-25 ε^-10).

Measured effects reported in the paper

Limitations

• Global convergence algorithm requires enumerating many vertices of the ambiguity set, which is costly.
• Experiments are small tabular simulations only (10 states, 5 actions).
• Results are restricted to tabular/finite state-action spaces and s-rectangular polyhedral sets.

Experimental setup

Metrics: norm of projected gradient vs sample complexity

Builds on: RL with general utility (Zhang et al. 2020) robust RL with s-rectangular ambiguity sets (Wiesemann et al. 2013) gradient descent ascent for nonconvex-concave minimax (Lin et al. 2020) variational policy gradient for convex RL normalized policy gradient with recursive momentum (Ben Hu et al.)

Closest papers by meaning

• A Reduction Framework for Distributionally Robust Reinforcement Learning under Average Reward (ICML 2025 · similarity 0.90)
• On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents (NeurIPS 2025 · similarity 0.88)
• Time-Constrained Robust MDPs (NeurIPS 2024 · similarity 0.88)
• Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
• Constrained Reinforcement Learning Under Model Mismatch (ICML 2024 · similarity 0.87)