Research map / Reinforcement learning / Policy gradient and actor-critic

On Corruption-Robustness in Performative Reinforcement Learning

AAAI 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Extends repeated retraining for performative RL to Huber ε-contaminated data via a robust OFTRL saddle-point solver and a coordinate-wise robust mean estimator, proving last-iterate convergence to an approximately performatively stable policy with error linear in √ε.

Problem

Performative reinforcement learning where the policy shifts the environment and the offline training data is adversarially corrupted under Huber ε-contamination, so repeated retraining may fail to converge to a performatively stable policy.

Approach

Combines robust optimistic FTRL for convex-concave saddle points under corrupted gradients with a problem-specific coordinate-wise robust mean estimator for the Lagrangian gradients, embedded in repeated retraining; proves an information-theoretic lower bound on the duality gap under bounded gradient noise.

Key results

• Robust repeated retraining converges under corruption while naive gradient averaging oscillates with magnitude scaling with the corruption magnitude Z (gridworld experiments, eps=0.01, averaged over 10 seeds).
• Theoretical last-iterate guarantee: output lies within distance 4·C̄(δ/N) of a performatively stable occupancy measure, with approximation error proportional to √ε (ignoring ε-independent terms).
• Robust OFTRL achieves O(1/T) duality gap and a lower bound shows the dependence on gradient noise norm is information-theoretically optimal.

Measured effects reported in the paper

Limitations

• Assumes s_i and a_i are uncorrupted; only next state and reward can be corrupted
• Experiments are limited to a small simulated gridworld with known sampling occupancy measure
• Approximation error tightness with respect to epsilon is left open

Experimental setup

Datasets: simulated gridworld transition samples

Benchmarks: 8x8 gridworld performative RL simulation

Metrics: normalised distance between successive occupancy measures d_

Baselines: naive gradient averaging repeated retraining

Builds on: performative RL (Mandal et al.) optimistic FTRL Huber contamination model Lagrangian minimax formulation of offline RL

Closest papers by meaning

• Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption (ICML 2024 · similarity 0.88)
• Towards Robust Offline Reinforcement Learning under Diverse Data Corruption (ICLR 2024 · similarity 0.88)
• Blending Imitation and Reinforcement Learning for Robust Policy Improvement (ICLR 2024 · similarity 0.87)
• Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
• State Entropy Regularization for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
• Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards (NeurIPS 2026 · similarity 0.86)