Research map / Reinforcement learning / Offline RL
Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies offline RL under random and adversarial corruption of states, actions, rewards and dynamics, finds IQL robust, and proposes Robust IQL (RIQL) with observation normalization, Huber loss and quantile Q ensembles to handle heavy-tailed dynamics corruption.
Problem
Offline RL learned from noisy or maliciously corrupted datasets (states, actions, rewards, dynamics) degrades significantly, and prior offline RL methods lack robustness to training-time data corruption.
Approach
Explains IQL's robustness via weighted imitation learning and a theoretical bound, then adds Huber loss for heavy-tailed Q targets under dynamics corruption and an ensemble quantile Q estimator that penalizes in-dataset corrupted samples, yielding RIQL.
Key results
- Under random corruption, RIQL averages 60.0 normalized score vs 43.6 for IQL, with 17.0% average degradation vs 31.2% for IQL.
- Under adversarial corruption, RIQL averages 56.2 vs 33.1 for IQL, with 22.0% degradation vs 46.0% for IQL.
- RIQL achieves 78.2 on Walker2d dynamics random corruption where IQL scores 17.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace IQL with RIQL, adding observation normalization, Huber loss, and quantile Q estimators | IQL · Random corruption (12 settings) | average score: 43.6 -> 60.0 | Helped |
| Replace IQL with RIQL, adding observation normalization, Huber loss, and quantile Q estimators | IQL · Adversarial corruption (12 settings) | average score: 33.1 -> 56.2 | Helped |
| Add observation normalization (ablation) | RIQL · Mixed attack (Walker2d and Hopper, medium-replay-v2) | average return | Helped |
| Add Huber loss (ablation) | RIQL · Mixed attack (Walker2d and Hopper, medium-replay-v2) | average return | Helped |
| Add quantile Q estimator (ablation) | RIQL · Mixed attack (Walker2d and Hopper, medium-replay-v2) | average return | Helped |
| Replace clipped double Q with a single Q function (ablation) | IQL · Most corruption settings except dynamics corruption | return | Hurt |
| Replace clipped double Q with a single Q function (ablation) | IQL · Dynamics corruption | return | Helped |
| Use observation normalization (ablation) | IQL · Diverse data attacks (four datasets) | average performance | Helped |
| Use Huber loss instead of quadratic loss for Q learning (ablation) | IQL · Dynamics attack | robustness | Helped |
| Use quantile Q estimators with an ensemble instead of the minimum-Q estimator | IQL · Dynamics corruption | return | Helped |
Surprising findings
- Behavior cloning shows only 0.4% average degradation under random corruption, though its absolute performance is low.
- The minimum operator in clipped double Q-learning hurts IQL under dynamics corruption, causing value explosion, while a single Q function is more stable there.
Limitations
- Observation corruption remains the most challenging setting for RIQL, indicating further improvements are needed.
- Experiments are limited to MuJoCo tasks (Halfcheetah, Walker2d, Hopper) with synthetic corruption.
- Theoretical guarantees rely on assumptions such as partial coverage and heavy-tailed noise with bounded (1+nu)-th moment.
Experimental setup
Datasets: D4RL medium-replay-v2 D4RL medium-expert-v2
Benchmarks: MuJoCo Halfcheetah MuJoCo Walker2d MuJoCo Hopper
Metrics: normalized score average degradation percentage
Baselines: BC EDAC MSG CQL SQL IQL
Builds on: IQL EDAC MSG Clipped Double Q-learning D4RL
Closest papers by meaning
- Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling (ICLR 2025 · similarity 0.94)
- Exact Policy Recovery in Offline RL with Both Heavy-Tailed Rewards and Data Corruption (AAAI 2024 · similarity 0.93)
- Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption (NeurIPS 2025 · similarity 0.93)
- Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions (NeurIPS 2024 · similarity 0.92)
- Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption (ICML 2024 · similarity 0.91)
- Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.