Research map / Reinforcement learning / Policy gradient and actor-critic

Risk-Conditioned Reinforcement Learning: A Generalized Approach for Adapting to Varying Risk Measures

AAAI 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GRIPS, a risk-conditioned SAC/TQC agent that adapts to arbitrary Weighted Value-at-Risk measures via a conditional adversarial auto-encoder risk proposal network and a non-crossing normalizing-flow quantile regressor.

Problem

Standard risk-sensitive RL agents are optimized for a single risk measure, and existing risk-conditioned methods (RCDSAC) only cover limited parameterized families of risk measures, so agents cannot adapt to varying risk preferences at deployment.

Approach

Represent any WV@R risk measure (as a quantile function) with a conditional AAE embedding, sample risk measures with a controlled optimism cut-off, and enforce monotone (non-crossing) quantile functions with a conditional bijective normalizing flow, so one actor-critic can be conditioned on arbitrary risk measures.

Key results

• On Donkey Car self-driving, GRIPS achieves a 16%-54% score increase over IQN across risk measures.
• On Stock-Hedge, GRIPS is 3.9%-54.0% higher than IQN; on Stock-Unhedge it ranges from -6.81% to +47%.
• In Mujoco with injected reward noise, GRIPS and RCDSAC are ~24% below IQN on HalfCheetah but outperform IQN on Walker2d.

Measured effects reported in the paper

Surprising findings

• Risk-conditioned GRIPS often outperforms IQN trained for the specific evaluation risk measure, especially when exploration is needed during training.
• Without the non-crossing quantile regressor, generated risk measures contain many monotonicity violations and risk-conditioned policies fail to generalize except for the neutral case.

Limitations

• Covers only WV@R risk measures; does not handle measures such as CMV or variance.
• Excessively optimistic risk measures are excluded, and performance drops in low-exploration environments such as HalfCheetah.
• Evaluation on a limited set of environments and risk measures.

Experimental setup

Datasets: MuJoCo HalfCheetah MuJoCo Hopper MuJoCo Walker2d MetaTrader stock simulator Donkey Car simulator AVC-Sparkfun map toy single-step environment

Benchmarks: MuJoCo HalfCheetah MuJoCo Hopper MuJoCo Walker2d MetaTrader Stock-Hedge MetaTrader Stock-Unhedge Donkey Car AVC-Sparkfun

Metrics: episode score under evaluation risk measure collision rate relative score w.r.t. IQN MSE of actions in toy env

Baselines: IQN RCDSAC

Builds on: IQN RCDSAC TQC SAC WGAN-GP

Closest papers by meaning

• Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.90)
• Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning (ICML 2025 · similarity 0.89)
• A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents (ICML 2025 · similarity 0.89)
• Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively (AAAI 2024 · similarity 0.88)
• Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation (ICML 2024 · similarity 0.88)
• Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.88)