Research map / Reinforcement learning / MDPs and planning
Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RS-DisRL, a general framework for risk-sensitive distributional RL with static Lipschitz risk measures and general function approximation, with model-based (RS-DisRL-M) and model-free (RS-DisRL-V) meta-algorithms achieving Õ(√K) regret via LSR and MLE analyses.
Problem
Sample-efficient risk-sensitive RL with static Lipschitz risk measures (covering ERM, CVaR, spectral risk) under general function approximation; prior theory was limited to tabular MDPs or specific risk measures.
Approach
Analyze risk-sensitive distributional RL on augmented MDPs (state augmented with cumulative reward) using CDF-based Bellman equations, and design least-squares-regression and maximum-likelihood estimators for both model-based and value-based function approximation, yielding Õ(√K) regret for static LRM.
Key results
- Model-based RS-DisRL-M with LSR achieves regret Õ(L(ρ) H D √K) and with MLE Õ(L(ρ) poly(H) D √K).
- Model-free RS-DisRL-V with LSR achieves Õ(L(ρ) poly(H) √(K D_cov dim_E)) and with MLE Õ(L(ρ) poly(H) √(K d_BE D_cov)).
- Claims the first Õ(√K) regret bounds for RSRL with static Lipschitz risk measures; the ERM bound avoids the e^{|γ|H} factor of prior work.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use distributional analysis to propagate Bellman error instead of the prior ERM analysis | RS-DisRL-M + M-Est-LSR · RS-DisRL with ERM | regret upper bound | Helped |
| Use MLE estimation with the augmented simulation lemma | RS-DisRL-M + M-Est-MLE · RS-DisRL with model-based function approximation | regret upper bound | Helped |
| Use LSR estimation in the general value function approximation framework | RS-DisRL-V + V-Est-LSR · RS-DisRL with general value function approximation | regret upper bound | Helped |
| Use MLE estimation in the general value function approximation framework | RS-DisRL-V + V-Est-MLE · RS-DisRL with general value function approximation | regret upper bound | Helped |
Limitations
- Regret bounds depend on structural complexity measures (eluder dimension, covering/bracketing numbers) that may be large; assumes agent knows the reward distribution.
- Model-free results require distributional Bellman completeness assumption for the value function class.
Experimental setup
Metrics: regret
Builds on: OMLE augmented MDP (static risk-sensitive RL) distributional Bellman equation eluder dimension
Closest papers by meaning
- Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.91)
- A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents (ICML 2025 · similarity 0.90)
- Beyond CVaR: Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning (ICML 2025 · similarity 0.90)
- Near-Optimal Distributionally Robust Reinforcement Learning with General $L_p$ Norms (NeurIPS 2024 · similarity 0.89)
- More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning (ICML 2024 · similarity 0.89)
- Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.