Research map / Reinforcement learning / Policy gradient and actor-critic
Provably Efficient Iterated CVaR Reinforcement Learning with Function Approximation and Human Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ICVaR-L, ICVaR-G and ICVaR-HF, provably sample-efficient algorithms for iterated CVaR RL with linear/general function approximation and human (comparison) feedback, with regret bounds and a linear-case lower bound.
Problem
Risk-sensitive episodic RL with an Iterated CVaR objective under function approximation (no simulator), extended to RL from human comparison feedback, where prior theory covered only tabular MDPs or risk-neutral settings.
Approach
Uses an epsilon-net approximation of the CVaR operator that is computable by taking a supremum over a finite grid, a ridge regression with CVaR-adapted features (psi of (x - V)^+), a refined elliptical potential lemma for general function classes, and a regret decomposition plus reward MLE over an infinite reward set for risk-sensitive RLHF.
Key results
- ICVaR-L regret upper bound of order sqrt(alpha^{-(H+1)}(d^2H^4+dH^6)K) with linear function approximation.
- Lower bound Omega(d sqrt(alpha^{-H+1} K)) for the linear ICVaR setting, claimed near-minimax in d and K.
- In synthetic experiments with 10 runs, ICVaR-L shows sublinear regret and lower regret than ICVaR-VI and LSVI for alpha in {0.15, 0.30}.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ICVaR-L with linear function approximation instead of tabular ICVaR-VI | ICVaR-RL · Risk MDP experiments (S = 20, d = 2; S = 40, d = 4; α ∈ {0.15, 0.30}) | cumulative regret | Helped |
| use ICVaR-L instead of risk-neutral LSVI | ICVaR-RL · Risk MDP experiments (S = 20, d = 2; S = 40, d = 4; α ∈ {0.15, 0.30}) | cumulative regret | Helped |
| use risk-neutral LSVI instead of ICVaR-L | Risk-sensitive linear RL · Risk MDP experiments (S = 20, d = 2; S = 40, d = 4; α ∈ {0.15, 0.30}) | cumulative regret | Hurt |
Surprising findings
- The paper's own regret bound exponent in alpha, alpha^{-(H+1)}, is exponentially large in horizon H, and the lower bound leaves a gap in alpha and H.
Limitations
- Experiments are small synthetic MDPs with only 10 runs and no real-world or LLM evaluation.
- Gap between upper and lower regret bounds in alpha and H remains open, as the authors note.
- Regret bounds scale exponentially in H through alpha^{-(H+1)}.
Experimental setup
Benchmarks: custom synthetic risk MDP (S=20,d=2; S=40,d=4)
Metrics: cumulative regret
Baselines: ICVaR-VI LSVI
Builds on: ICVaR-RM UCRL-VTR / linear mixture MDP algorithms Xiong et al. general function approximation RL BTL preference model
Closest papers by meaning
- Risk-Sensitive Reward-Free Reinforcement Learning with CVaR (ICML 2024 · similarity 0.91)
- Provably Efficient CVaR RL in Low-rank MDPs (ICLR 2024 · similarity 0.91)
- A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents (ICML 2025 · similarity 0.90)
- Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation (ICML 2024 · similarity 0.89)
- Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
- Safe Reinforcement Learning with Instantaneous Constraints: The Role of Aggressive Exploration (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.