Research map / Reinforcement learning / Policy gradient and actor-critic

Risk-Sensitive Policy Optimization via Predictive CVaR Policy Gradient

ICML 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Variance-reduced policy gradients

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes predictive CVaR policy gradient (PCVaR), which reweights per-step costs by learned predictive tail probabilities so risk-neutral policy gradient methods can optimize CVaR using all trajectories.

Problem

Policy optimization under the CVaR risk objective is sample-inefficient because existing CVaR policy gradient methods use only the worst q-fraction of sample trajectories.

Approach

Reformulate CVaR as a time-separable risk-neutral objective weighted by predictive tail probabilities (the probability that the total cost exceeds the VaR threshold given history), learn these weights via a separate supervised model, and plug the reweighted costs into standard risk-neutral policy gradient estimators.

Key results

• In continuous Blackjack, PCVaR gradient variance at the CVaR-optimal point is 6.71 versus 24.01 (NCVaR) and 24.67 (GCVaR), and PCVaR converges to the CVaR-optimal theta ≈14.2 faster.
• In pair trading (q=0.2), PCVaR achieves higher and more stable cumulative returns than the baseline policy gradient algorithms over the test period.
• Theoretical results show consistency of the predictive tail probability estimator and η estimator under stated step-size conditions, and unbiasedness of the reweighted gradient estimators.

Measured effects reported in the paper

Limitations

• Restricted to Markov policies; non-anticipating policies not covered.
• No convergence rate or global optimality guarantees.
• Stability analysis relies on the assumption that the policy changes slowly relative to the prediction model's estimation.

Experimental setup

Datasets: Tiingo end-of-day stock data (990 days, 2015-2018)

Benchmarks: continuous Blackjack game pair trading with Tiingo stock data

Metrics: gradient estimate variance policy parameter convergence cumulative return (%)

Baselines: GCVaR (Tamar et al., 2015b) NCVaR (naive CVaR policy gradient) risk-neutral policy gradient

Builds on: Tamar et al. 2015 CVaR policy gradient variational representation of CVaR (Rockafellar and Uryasev) Rao-Blackwellization Huang et al. 2021 CVaR reweighting

Closest papers by meaning

• Return Capping: Sample Efficient CVaR Policy Gradient Optimisation (ICML 2025 · similarity 0.90)
• Risk-Sensitive Reward-Free Reinforcement Learning with CVaR (ICML 2024 · similarity 0.87)
• A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents (ICML 2025 · similarity 0.86)
• Improving Value Estimation Critically Enhances Vanilla Policy Gradient (ICML 2025 · similarity 0.86)
• Provably Efficient CVaR RL in Low-rank MDPs (ICLR 2024 · similarity 0.86)
• Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback (ICLR 2024 · similarity 0.85)