Research map / Reinforcement learning / Policy gradient and actor-critic
Return Capping: Sample Efficient CVaR Policy Gradient Optimisation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Return Capping, a CVaR policy gradient reformulation that caps trajectory returns at an approximated VaR instead of discarding trajectories, improving sample efficiency over CVaR-PG and CVaR-PPO baselines.
Problem
Policy-gradient optimisation of static CVaR discards most sampled trajectories (all but the bottom alpha proportion), making it very sample-inefficient, especially for small alpha, and blind to high-return successes and prone to vanishing gradients.
Approach
Keep all trajectories but truncate returns above a cap C; proves that maximising expected capped return with C set to VaR_alpha of the optimal CVaR policy yields the CVaR-optimal policy. The cap is tracked online via a smoothed VaR estimate with a minimum-cap floor, and rewards are adjusted so the capped return can be optimised with reward-to-go PPO.
Key results
- On the Betting Game (CVaR alpha=0.2), Return Capping outperforms CVaR-PPO and CVaR-PG, with CVaR-PG converging to an overly conservative policy with CVaR near zero.
- In the continuous Guarded Maze (CVaR alpha=0.05), all six Return Capping seeds converged to the CVaR-optimal policy, while no CVaR-PG runs reached the goal and only two of six CVaR-PPO seeds found the CVaR-optimal policy.
- In the discrete Guarded Maze, Return Capping converges in approximately half as many environment steps as CVaR-PPO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use return capping instead of discarding trajectories | CVaR policy-gradient training · Betting Game, CVaR α=0.2 | CVaR performance | Helped |
| Use return capping instead of trajectory-discarding CVaR methods | CVaR policy-gradient training · Autonomous Vehicle, CVaR α=0.05 | training speed to convergence | Mixed |
| Use return capping instead of CVaR-PG or CVaR-PPO | CVaR policy-gradient training · Guarded Maze continuous, CVaR α=0.05 | convergence to CVaR-optimal policy | Helped |
| Use return capping instead of CVaR-PPO | CVaR policy-gradient training · Guarded Maze discrete, CVaR α=0.2 | environment steps to convergence | Helped |
| Use return capping instead of CVaR-PPO | CVaR policy-gradient training · Lunar Lander, CVaR α=0.2 | policy performance | Helped |
| Initialize the minimum cap using different policy-return estimates (ablation) | Return Capping · Betting Game, CVaR α=0.2 | CVaR performance | Mixed |
| Initialize the minimum cap using different policy-return estimates (ablation) | Return Capping · Autonomous Vehicle, CVaR α=0.05 | training performance | No clear effect |
| Initialize the minimum cap using different policy-return estimates (ablation) | Return Capping · Guarded Maze continuous and discrete | final policy quality | Mixed |
| Set the minimum cap to the CVaR of the conservative policy (ablation) | Return Capping · Lunar Lander, CVaR α=0.2 | convergence to the optimal CVaR policy | Hurt |
| Use MIX to combine the risk-neutral policy with a learned risk-averse policy | MIX baseline · Lunar Lander, CVaR α=0.2 | learning the risk-averse policy | Hurt |
Surprising findings
- In the Guarded Maze domains, the best final policy came from a low (not high) minimum cap, opposite to the other domains where setting the cap to VaR of the optimal CVaR policy worked best.
- In Lunar Lander, initialising the minimum cap from the conservative do-nothing policy did not consistently yield the CVaR-optimal policy.
Limitations
- Performance is sensitive to the minimum cap initialisation in some environments (notably the guarded maze domains).
- Evaluated on a limited set of relatively simple environments.
- Minimum cap validity relies on assumptions (e.g., a cap marginally above the conservative policy's CVaR is assumed valid), and the risk-neutral-based minimum cap may be non-trivial to determine.
Experimental setup
Datasets: Betting Game Autonomous Vehicle navigation Continuous Guarded Maze Discrete Guarded Maze Lunar Lander (OpenAI Gym, modified)
Benchmarks: Betting Game Autonomous Vehicle Continuous Guarded Maze Discrete Guarded Maze Lunar Lander
Metrics: CVaR of return return vs environment steps convergence robustness across seeds
Baselines: PPO (expected value) CVaR-PG CVaR-PPO MIX
Builds on: PPO CVaR policy gradient (Tamar et al.) MIX (policy mixing) Soft Risk
Closest papers by meaning
- Boosting CVaR Policy Optimization with Quantile Gradients (ICML 2026 · similarity 0.91)
- Risk-Sensitive Policy Optimization via Predictive CVaR Policy Gradient (ICML 2024 · similarity 0.90)
- Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy (AAAI 2020 · similarity 0.87)
- Improving Value Estimation Critically Enhances Vanilla Policy Gradient (ICML 2025 · similarity 0.84)
- Risk-Sensitive Reward-Free Reinforcement Learning with CVaR (ICML 2024 · similarity 0.84)
- Provably Efficient CVaR RL in Low-rank MDPs (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.