Research map / Reinforcement learning / Offline RL
Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that Decision Transformer is inherently robust to training-time data corruption in limited offline RL data and proposes Robust Decision Transformer (RDT), adding embedding dropout, Gaussian weighted learning, and iterative data correction.
Problem
Offline RL learned from corrupted (noisy or adversarial) limited datasets; temporal-difference offline RL methods degrade sharply under data corruption, and it is unclear whether sequence modeling can handle this.
Approach
Vanilla Decision Transformer is already robust to data corruption; RDT strengthens it with dimension-wise embedding dropout on token embeddings, Gaussian weighting of action and reward losses by per-sample prediction error, and iterative z-score-based correction of corrupted actions and rewards using model predictions, plus reward prediction.
Key results
- Under random corruption, RDT averages 57.8 normalized score versus 45.1 for DT over all tasks and attack types.
- Under adversarial corruption, RDT averages 54.3 versus 41.7 for DT, and improves over DT by 105% in adversarial action corruption.
- RDT achieves the best score in 24 of 27 random-corruption settings and reports a 28.2% overall improvement over DT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RDT's embedding dropout, Gaussian weighted learning, and iterative data correction | Decision Transformer · Random corruption: all tasks | normalized score: 45.1 -> 57.8 | Helped |
| Add RDT's three robust techniques | Decision Transformer · Adversarial action corruption: all tasks | normalized score: 22.1 -> 45.3 | Helped |
| Add Gaussian weighted learning (ablation) | DT(RP) · Reward attack | normalized score | Helped |
| Add embedding dropout (ablation) | DT(RP) · Data corruption scenarios | normalized score | Helped |
| Add iterative data correction (ablation) | DT(RP) · Data corruption scenarios | normalized score | Helped |
| Add reward prediction (ablation) | Decision Transformer · Random state and action corruption | normalized score | Helped |
| Add reward prediction (ablation) | Decision Transformer · Reward corruption | normalized score | Hurt |
| Use a moderate embedding dropout probability such as 0.1 (ablation) | RDT · Random action corruption | normalized score | Helped |
| Increase embedding dropout rate to 0.5 or higher (ablation) | RDT · Random action corruption | normalized score | Hurt |
| Use a larger transformer block count (ablation) | Decision Transformer · MuJoCo and Kitchen | normalized score | Helped |
| Remove return-to-go inputs (ablation) | Decision Transformer · MuJoCo and Kitchen | normalized score | Mixed |
| Add a transformer backbone (ablation) | UWMSG · Offline RL benchmarks | overall performance | No clear effect |
Surprising findings
- Vanilla Decision Transformer is already more robust to data corruption than temporal-difference offline RL methods (CQL, UWMSG, RIQL) when data is limited.
- Predicting rewards in DT helps under state and action corruption but hurts under reward corruption.
Limitations
- Performance declines under state corruption at high corruption rates, attributed to distribution shift from corrupted states.
- Iterative data correction only corrects actions and rewards; state correction is left for future work.
- Experiments use simulated corruption on small subsampled D4RL datasets rather than real-world noisy data.
Experimental setup
Datasets: D4RL MuJoCo medium-replay-v2 D4RL Kitchen D4RL Adroit expert-v0
Benchmarks: MuJoCo (10%) Kitchen Adroit (1%) MuJoCo full dataset
Metrics: normalized score average normalized return across seeds
Baselines: BC RBC DeFog CQL UWMSG RIQL DT RORL
Builds on: Decision Transformer Implicit Q-Learning CORL
Closest papers by meaning
- Towards Robust Offline Reinforcement Learning under Diverse Data Corruption (ICLR 2024 · similarity 0.94)
- Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption (NeurIPS 2025 · similarity 0.91)
- ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
- Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.90)
- Exact Policy Recovery in Offline RL with Both Heavy-Tailed Rewards and Data Corruption (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.