Research map / Reinforcement learning / Offline RL
Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReDOR, which selects a reduced offline RL dataset by casting TD-gradient matching as a weakly submodular problem solved with modified orthogonal matching pursuit (trajectory-based, multi-round, return-constrained), and trains TD3+BC/IQL on the subset.
Problem
Offline RL training on large pre-collected datasets is computationally costly, and suboptimal data can worsen distribution shift; the paper asks how to select an offline data subset that maintains or improves performance while reducing training cost.
Approach
Frame dataset selection as gradient approximation of the TD loss, show the actor-critic objective is weakly submodular, and adapt OMP with empirical-return targets, trajectory-level gradient basis vectors, multi-round selection, and a top-m% return constraint to handle changing offline RL targets.
Key results
- On D4RL (hard) MuJoCo tasks, ReDOR outperforms KRLS, Log-Det and BlockGreedy on all nine listed datasets, e.g., Walker2d-medium 80.5 vs 53.3 (BlockGreedy).
- On AntMaze with IQL, ReDOR beats the complete dataset on all six tasks, e.g., antmaze-medium-diverse 84.9 vs 70.0.
- Selection on datasets of about 1M transitions takes roughly 8 minutes on an RTX 3090 according to the reported complexity table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select a reduced dataset using ReDOR instead of supervised-learning reduction baselines | TD3+BC · D4RL Hard Mujoco tasks | normalized score | Helped |
| Select a reduced dataset using ReDOR instead of the complete dataset | IQL · Antmaze tasks | normalized score | Helped |
| Replace empirical-return targets with the standard target Q function (ablation) | ReDOR · D4RL Hard Walker2d-medium | normalized score | Hurt |
| Use a single data-selection round instead of multi-round selection (ablation) | ReDOR · D4RL Hard Walker2d-medium | normalized score: over 80% | Hurt |
| Replace empirical-return targets with the standard target Q function (ablation) | ReDOR · D4RL Hard Walker2d-expert | normalized score: 95% | Hurt |
| Replace empirical-return targets with the standard target Q function (ablation) | ReDOR · D4RL Hard Halfcheetah tasks | normalized score | No clear effect |
| Use a single data-selection round instead of multi-round selection (ablation) | ReDOR · D4RL Hard Halfcheetah tasks | normalized score | No clear effect |
Surprising findings
- Reduced subsets selected by ReDOR outperform training on the complete dataset in AntMaze tasks, suggesting that data reduction can improve offline RL performance.
- Supervised-learning data selection methods (KRLS, Log-Det, BlockGreedy) do not transfer well to offline RL because of evolving targets.
Limitations
- Theoretical analysis assumes uniformly bounded gradients, which is not rigorously guaranteed in practice.
- Theory is based on classic TD loss, while the method uses fixed empirical-return targets instead.
- Mainly evaluated on D4RL MuJoCo and AntMaze with TD3+BC and IQL; real-world robotics is left for future work.
Experimental setup
Datasets: D4RL MuJoCo (hopper, walker2d, halfcheetah) D4RL AntMaze D4RL (hard) with added low-quality data
Benchmarks: D4RL D4RL (hard) AntMaze
Metrics: normalized return computational time
Baselines: Complete Dataset Prioritized Random KRLS Log-Det BlockGreedy
Builds on: TD3+BC IQL orthogonal matching pursuit MOPO prioritized experience replay
Closest papers by meaning
- A Minimalist Approach to Offline Reinforcement Learning (NeurIPS 2021 · similarity 0.93)
- Efficient Online Reinforcement Learning with Offline Data (ICML 2023 · similarity 0.92)
- Leveraging Offline Data in Online Reinforcement Learning (ICML 2023 · similarity 0.92)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Horizon Reduction Makes RL Scalable (NeurIPS 2025 · similarity 0.90)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.