Research map / Reinforcement learning / Offline RL
Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RPEX, which adds inverse-probability-weighted (IPW) terms to the policy-expansion action selection so that a frozen offline policy and a learnable online policy are mixed robustly under offline and online data corruption in O2O RL.
Problem
Offline-to-online RL degrades when offline and online data are corrupted (states, actions, rewards, dynamics), because corruption makes the policy heavy-tailed and hinders efficient online exploration.
Approach
Data corruption makes the policy heavy-tailed; adding an IPW regularization term (Q-V divided by the action probability, clipped) to PEX's composite policy selection penalizes low-quality, rarely-chosen actions and improves exploration efficiency.
Key results
- RPEX (RIQL-based) achieves average O2O score 75.16 vs 66.6 for RIQL-PEX and 54.9 for RIQL on medium-replay tasks, with average improvement of 28.7% over RIQL's offline baseline.
- IQL-RPEX achieves average improvement 32.7% vs 13.1% for IQL and 20.3% for IQL-PEX.
- Paper reports roughly 13.6% O2O improvement of RPEX over vanilla PEX, at about 8.1% extra training time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add IPW to PEX | RIQL-PEX (RPEX) · Medium-Replay tasks, random corruption | average O2O score: 66.6 -> 75.16 | Helped |
| Add IPW to PEX | IQL-PEX (IQL-RPEX) · Medium-Replay tasks, random corruption | average O2O score: 53.5 -> 59.1 | Helped |
| Increase critic UTD from 1 to 4 while actor UTD remains 1 (ablation) | RPEX · Walker2d-MR, dynamics corruption | normalized score: 80 → 92.2; 80 → 92.6 | Helped |
| Increase critic UTD from 1 to 4 while actor UTD remains 1 (ablation) | RPEX · Walker2d-MR, action corruption | normalized score: 85.9 → 118.9; 85.9 → 96.7 | Hurt |
| Replace clip range (-10000, 100) with (-100, 100) (ablation) | RPEX · Walker2d-MR, dynamics corruption | normalized score: 80 → 92.2; 80 → 90.8 | Hurt |
| Replace clip range (-10000, 100) with (0, 100) (ablation) | RPEX · Walker2d-MR, dynamics corruption | normalized score: 80 → 92.2; 80 → 62.1 | Hurt |
| Replace clip range (-10000, 100) with (-10000, 10000) (ablation) | RPEX · Walker2d-MR, dynamics corruption | normalized score: 80 → 92.2; 80 → 96.7 | Helped |
| Replace clip range (-10000, 100) with (-100, 100) (ablation) | RPEX · Walker2d-MR, action corruption | normalized score: 85.9 → 118.9; 85.9 → 108.4 | Hurt |
| Replace clip range (-10000, 100) with (0, 100) (ablation) | RPEX · Walker2d-MR, action corruption | normalized score: 85.9 → 118.9; 85.9 → 62.67 | Hurt |
| Replace clip range (-10000, 100) with (-10000, 10000) (ablation) | RPEX · Walker2d-MR, action corruption | normalized score: 85.9 → 118.9; 85.9 → 105.53 | Hurt |
| Apply state normalization during O2O fine-tuning (ablation) | O2O methods · Hopper-MR, action and dynamics attacks | O2O performance | Helped |
| Use a deterministic rather than stochastic policy (ablation) | O2O methods · Hopper-MR, action and dynamics attacks | O2O performance | Hurt |
Surprising findings
- Policy kurtosis of RIQL under attack is higher than IQL, suggesting RIQL reduces Q-target heavy-tailedness at the cost of more policy heavy-tailedness.
- A deterministic offline policy can degrade O2O performance, unlike in the offline setting.
Limitations
- IPW requires exact action probabilities, which may hinder integration with probability-agnostic models such as diffusion models.
- Observation and dynamics corruption on AntMaze remain hard; RPEX still struggles to improve performance there.
Experimental setup
Datasets: D4RL halfcheetah-medium-replay D4RL walker2d-medium-replay D4RL hopper-medium-replay D4RL antmaze-large D4RL antmaze-diverse
Benchmarks: D4RL halfcheetah-medium-replay D4RL walker2d-medium-replay D4RL hopper-medium-replay D4RL antmaze-large D4RL antmaze-diverse
Metrics: normalized score average O2O score average improvement percentage
Baselines: IQL IQL-PEX RIQL RIQL-PEX WSRL Cal-QL
Builds on: Policy Expansion (PEX) Implicit Q-learning (IQL) Robust IQL (RIQL) AlignIQL EDAC
Closest papers by meaning
- Towards Robust Offline Reinforcement Learning under Diverse Data Corruption (ICLR 2024 · similarity 0.93)
- Exact Policy Recovery in Offline RL with Both Heavy-Tailed Rewards and Data Corruption (AAAI 2024 · similarity 0.91)
- Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling (ICLR 2025 · similarity 0.91)
- ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.89)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.