Research map / Reinforcement learning / Offline RL
A Clean Slate for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a taxonomy and bandit-based evaluation protocol that quantifies online tuning budgets for offline RL, provides clean JAX single-file reimplementations, and introduces the Unifloral unified hyperparameter space from which TD3-AWR and MoBRAC are derived.
Problem
Offline RL comparisons are confounded by ambiguous problem definitions, undocumented online hyperparameter tuning, and inconsistent, entangled reference implementations.
Approach
Define a taxonomy of offline RL variants and a UCB-bandit evaluation protocol that reports performance against the number of online evaluation episodes used for tuning; unify model-free and model-based offline RL algorithms into one hyperparameter space (Unifloral) implemented in minimal, consistent single-file JAX code, and use it to construct TD3-AWR and MoBRAC.
Key results
- Speed-ups of 131.5x vs OfflineRL-Kit and 74.8x vs CORL in average training time.
- TD3-AWR strictly dominates ReBRAC on 6 of 9 datasets and IQL on 7 datasets.
- MoBRAC outperforms other model-based methods on 6 of 9 datasets and ties MOPO on 3 others.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ReBRAC's behavior-cloning actor term with advantage-weighted regression | ReBRAC, producing TD3-AWR · D4RL datasets | performance curve dominance: 6 out of 9 datasets | Helped |
| Replace ReBRAC's behavior-cloning actor term with advantage-weighted regression | ReBRAC, producing TD3-AWR · D4RL datasets | performance curve dominance: 7 datasets | Helped |
| Use ReBRAC as the policy optimizer with synthetic rollouts from a MOPO world model | MOPO · D4RL datasets | performance versus other model-based methods: 6 out of 9 datasets | Helped |
| Use ReBRAC as the policy optimizer with synthetic rollouts from a MOPO world model | MOPO · D4RL datasets | performance versus other model-based methods: 3 others | Mixed |
| Reimplement offline RL algorithms in end-to-end compiled JAX | OfflineRL-Kit implementations · HalfCheetah medium-expert, 1M update steps | training speed: 131.5× | Helped |
| Reimplement offline RL algorithms in end-to-end compiled JAX | CORL implementations · HalfCheetah medium-expert, 1M update steps | training speed: 74.8 × | Helped |
Surprising findings
- Model-based methods MOPO, MOReL, and COMBO rank no higher than 6th of 10 and fail to beat BC on non-locomotion datasets.
- Distractor policies cause the bandit's performance to dip as more arms are pulled, contrary to expectation of monotone improvement.
Limitations
- Evaluation uses a fixed K=8 policy subsample and a limited set of recommended datasets.
- Model-based methods use hard-coded termination functions from the target environment, an unrealistic assumption.
Experimental setup
Datasets: D4RL
Benchmarks: D4RL
Metrics: normalized episodic return best-arm true return vs number of bandit pulls
Baselines: BC IQL ReBRAC SAC-N EDAC CQL TD3-BC MOPO MOReL COMBO DT OfflineRL-Kit
Builds on: TD3-BC ReBRAC IQL MOPO EDAC
Closest papers by meaning
- A Minimalist Approach to Offline Reinforcement Learning (NeurIPS 2021 · similarity 0.95)
- Efficient Online Reinforcement Learning with Offline Data (ICML 2023 · similarity 0.94)
- Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol (NeurIPS 2025 · similarity 0.92)
- Simple Ingredients for Offline Reinforcement Learning (ICML 2024 · similarity 0.92)
- Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization (ICLR 2024 · similarity 0.92)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.