Research map / Reinforcement learning / Policy gradient and actor-critic

Risk-sensitive control as inference with Rényi divergence

NeurIPS 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that control as inference with Rényi divergence variational inference equals log-probability regularized risk-sensitive control, derives risk-sensitive policy gradient and soft actor-critic, and proposes a Rényi-entropy-regularized risk-sensitive generalization of MaxEnt control.

Problem

Control as inference (CaI) uses KL divergence, so it is unclear how the risk-sensitivity implied by replacing KL with Rényi divergence affects the optimal policy and what control objective CaI with Rényi divergence optimizes.

Approach

Replace the KL divergence in CaI with the Rényi divergence D_{1+η}, showing the resulting variational problem equals an exponential-utility risk-sensitive control problem with log-probability regularization; the optimal policy is a Gibbs distribution over a Q-function satisfying a soft Bellman equation. Also introduces a Rényi-entropy-regularized risk-sensitive MaxEnt control via a duality lemma b

Key results

• RCaI is shown equivalent to LP-regularized risk-sensitive control, and as the risk parameter vanishes it recovers risk-neutral CaI and RL.
• On Pendulum-v1 with pendulum length changed from 1.0 to 1.5 without retraining, the policy trained with eta=0.02 showed less performance degradation than SAC (eta=0).
• Learning fails for |eta| larger than 0.03 due to numerical instability in the risk-sensitive SAC.

Measured effects reported in the paper

Surprising findings

• Learning with risk-sensitive SAC fails for |eta| > 0.03 because of numerical instability.
• More risk-seeking policies (eta<0) were found to be less robust to system perturbations.

Limitations

• Risk-sensitive SAC is numerically unstable for large |eta|, depending on cost scale.
• Experiments are limited to a single Pendulum-v1 environment.
• Choice of the Rényi order parameter is not addressed.

Experimental setup

Datasets: Pendulum-v1

Benchmarks: Pendulum-v1

Metrics: average episode cost

Baselines: SAC

Builds on: control as inference maximum entropy control soft actor-critic variational renyi bound REINFORCE

Closest papers by meaning

• A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents (ICML 2025 · similarity 0.88)
• Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation (ICML 2024 · similarity 0.87)
• Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.86)
• Risk-Conditioned Reinforcement Learning: A Generalized Approach for Adapting to Varying Risk Measures (AAAI 2024 · similarity 0.86)
• Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback (ICLR 2024 · similarity 0.86)
• Provably Efficient Partially Observable Risk-sensitive Reinforcement Learning with Hindsight Observation (ICML 2024 · similarity 0.86)