Research map / Reinforcement learning / MDPs and planning
Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends model-based RL to adversarial transition corruption, proposing CR-OMLE (online, regret Õ(√T + C)) and CR-PMLE (offline) using TV-based information-ratio uncertainty weights on MLE.
Problem
Model-based RL is vulnerable to adversarial corruption of transition dynamics; existing corruption-robust RL results cover only model-free methods, and uncertainty weighting does not transfer to MLE-based model learning in either online or offline settings.
Approach
Weight the MLE log-likelihood samples by a truncated TV-based information ratio (uncertainty) so that corruption terms are bounded by uncertainty times corruption, integrated into optimistic (online) and pessimistic (offline) MLE with a confidence set.
Key results
- CR-OMLE achieves regret Õ(H log B sqrt(T log|M| ED) + CH·ED), i.e. Õ(√T + C) in the linear case, with a matching Ω(HdC) lower bound on the corruption term for linear MDPs.
- CR-PMLE offline achieves suboptimality Õ(H log B sqrt(log|M|/(T Cov)) + CH/(T Cov)) under uniform coverage, with lower bound Ω(C/(T Cov)) matching up to H.
- Claimed to be the first corruption-robust model-based RL algorithms with provable guarantees, and the first provable model-based offline RL algorithm even without corruption.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TV-based uncertainty weights to optimistic MLE | CR-OMLE · Online episodic model-based RL under transition corruption | regret: \tcO\Big(H\log B\sqrt{T\log|\mathcal{M}|\ED(\mathcal{M},\sqrt{\lambda/T})} + CH\cdot\ED(\mathcal{M},\sqrt{\lambda/T})\Bi | Helped |
| Add TV-based uncertainty weights to pessimistic MLE | CR-PMLE · Offline model-based RL with uniform data coverage and transition corruption | suboptimality: \tcO\bigg(H\log B\sqrt{\frac{\log |\mathcal{M}|}{T\mathrm{Cov}(\mathcal{M},\mathcal{D})}} + \frac{CH}{T\mathrm{Cov}(\mat | Helped |
| Increase the predefined corruption tolerance threshold \bar{C} | CR-OMLE with unknown corruption level · Online episodic model-based RL | regret: \tcO\Big(H\log B\sqrt{T\log|\mathcal{M}|\ED(\mathcal{M},\sqrt{\lambda/T})} + \bar C H\cdot\ED(\mathcal{M},\sqrt{\lambda/ | Mixed |
| Require uniform data coverage | CR-PMLE · Offline model-based RL | suboptimality: \tcO\bigg(H\log B\sqrt{\frac{\log |\mathcal{M}|}{T\mathrm{Cov}(\mathcal{M},\mathcal{D})}} + \frac{CH}{T\mathrm{Cov}(\mat | Mixed |
Limitations
- Computationally inefficient version-space style confidence sets; practical algorithm via bootstrapped uncertainty is left as future work.
- Requires knowledge of corruption level C for the parameter alpha; unknown-C variant gives weaker guarantees.
- Offline lower bound matches only up to a factor of H; analysis restricted to finite model classes and assumes known rewards.
Experimental setup
Metrics: regret suboptimality
Builds on: OMLE uncertainty weighting for corruption-robust linear bandits/RL eluder dimension pessimistic offline RL
Closest papers by meaning
- Towards Robust Offline Reinforcement Learning under Diverse Data Corruption (ICLR 2024 · similarity 0.91)
- Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Exact Policy Recovery in Offline RL with Both Heavy-Tailed Rewards and Data Corruption (AAAI 2024 · similarity 0.90)
- Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling (ICLR 2025 · similarity 0.89)
- Corruption-Robust Linear Bandits: Minimax Optimality and Gap-Dependent Misspecification (NeurIPS 2024 · similarity 0.89)
- Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.