Research map / Reinforcement learning / Policy gradient and actor-critic

Ergodic Risk Measures: Towards a Risk-Aware Foundation for Continual Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Temporal-difference and discounting theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a risk-aware formalism for continual RL, shows classical static and nested risk measures are incompatible with continual learning, introduces ergodic risk measures, and illustrates them with a tabular CVaR Q-learning case study on red-pill blue-pill tasks.

Problem

Continual RL has been studied almost exclusively under risk-neutral objectives; classical risk measures (static and nested) are not compatible with the stability-plasticity requirements of continual learning.

Approach

Defines risk-aware observations and continual risk-aware RL, proposes feasibility and plasticity axioms, and introduces ergodic risk measures: non-nested dynamic risk measures whose one-step assessments can be accurately computed from a finite recent window, realized via the time-averaged CVaR objective under average-reward ergodicity assumptions.

Key results

• In the tau-RPBP task (tau changes from 0.9 to 0.1 at t=50,000), the tabular agent first stays in the blue world and then switches to the red world as its risk attitude becomes risk-averse (over 50 runs, 95% CI).
• In the S-RPBP task with three state-reward regimes, the agent adapts to find the state with the better CVaR after each change (over 10 runs, 95% CI).
• Theorem shows the average-reward CVaR objective under an ergodicity-like assumption corresponds to an ergodic risk measure satisfying both axioms.

Limitations

• Case study is limited to small tabular RPBP environments with a single algorithm.
• Empirical evidence is illustrative rather than a benchmark comparison against existing baselines.
• Axioms and theorem rely on ergodicity-like (unichain/communicating) assumptions that may not hold in practice.

Experimental setup

Datasets: red-pill blue-pill task (simulated, tau-RPBP and S-RPBP variants)

Benchmarks: tau-RPBP S-RPBP

Metrics: rolling percent of time in blue world state rolling reward CVaR

Builds on: RED CVaR Q-learning continual RL framework (plasticity definition) dynamic risk measures average-reward MDP Birkhoff ergodic theorem

Closest papers by meaning

• Risk-Conditioned Reinforcement Learning: A Generalized Approach for Adapting to Varying Risk Measures (AAAI 2024 · similarity 0.88)
• Position: Lifetime tuning is incompatible with continual reinforcement learning (ICML 2025 · similarity 0.87)
• Continual Reinforcement Learning by Planning with Online World Models (ICML 2025 · similarity 0.87)
• Autonomous Option Invention for Continual Hierarchical Reinforcement Learning and Planning (AAAI 2025 · similarity 0.86)
• On-Policy Algorithms for Continual Reinforcement Learning (Student Abstract) (AAAI 2025 · similarity 0.86)
• Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning (ICML 2024 · similarity 0.86)