Research map / Reinforcement learning / Offline RL
Simple Ingredients for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MOOD, a DMC-based offline RL testbed with heterogeneous mixed-objective data, shows TD3+BC, AWAC, and IQL degrade when diverse data is merged, and finds that scaling up network size (plus evaluation sampling and ensembles) closes the gap.
Problem
Offline RL algorithms fail when the buffer mixes data from heterogeneous tasks/behaviors, even when each subset alone is sufficient for learning the target task.
Approach
Builds a mixed-objective offline testbed (MOOD) and tests hypotheses (over-conservatism, network scale, epistemic uncertainty, advantage-weighting variance); finds network scale is the key factor, and proposes evaluation sampling and advantage sampled actor critic (ASAC) as auxiliary remedies.
Key results
- On MOOD mixed-objective data with shallow networks, IQL drops about 26.5% and AWAC about 28.7% versus same-objective data; humanoid drops more than 50% for all offline algorithms.
- Scaling to large simple MLPs (5-layer actor, 3-layer critic) lets IQL, AWAC and TD3+BC reach totals of about 370-373 on mixed MOOD datasets (vs ~355-370 for same-objective), closing the gap.
- On D4RL antmaze-v0, IQL with large modern architecture reaches 463.1 total, above the reported state-of-the-art baselines (e.g., XQL 420.9, DQL 417.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add data from different tasks to the offline buffer | Offline RL methods with shallow networks · MOOD mixed-objective versus same-objective datasets | normalized return: 360.3 -> 284.8 | Hurt |
| Use a large network instead of a shallow network | Offline RL methods · MOOD mixed-objective datasets | normalized return | Helped |
| Use a large modern architecture instead of a large simple architecture (ablation) | Offline RL methods · MOOD mixed-objective datasets | normalized return: 373.6 -> 370.0 | No clear effect |
| Add advantage sampling in place of AWAC's weighted importance sampling estimator (ablation) | AWAC · MOOD mixed-objective datasets | normalized return | No clear effect |
| Increase the critic ensemble from 2 to 5 critics (ablation) | Offline RL methods · MOOD mixed-objective datasets | normalized return: 373.0 -> 373.6 | Helped |
| Use evaluation sampling at test time (ablation) | Offline RL methods · MOOD mixed-objective Quadruped and Humanoid | normalized return | Mixed |
| Use evaluation sampling at test time (ablation) | Offline RL methods · D4RL Antmaze versus Locomotion datasets | performance | Mixed |
| Increase network scale | IQL · D4RL Antmaze-v0 | performance total: 443.2 -> 463.1 | Helped |
| Train with standard uniform sampling rather than return-proportional trajectory sampling | Large-network offline RL methods · D4RL Locomotion-0.1-v2 | performance total | No clear effect |
| Add diverse data | TD3 · MOOD mixed-objective datasets | normalized return | Helped |
Surprising findings
- Simply increasing network size, with standard uniform sampling, matches the sophisticated return-weighted sampling strategy previously proposed for unbalanced locomotion data.
- Plain TD3 improves with more diverse data, while offline methods degrade, and the modern residual architecture gave no clear advantage over a simple large MLP.
Limitations
- Evaluation sampling hurts on humanoid mixed-objective data, so the test-time remedy is not uniformly beneficial.
- Gains from ensembles of critics are marginal and come with extra compute cost.
- Study is restricted to DMC-based environments and three TD3-derived algorithms.
Experimental setup
Datasets: MOOD (DeepMind Control mixed-objective datasets) D4RL locomotion-v2 D4RL antmaze-v0 D4RL locomotion-0.1-v2 (unbalanced) ExoRL
Benchmarks: MOOD D4RL locomotion-v2 D4RL antmaze-v0 D4RL locomotion-0.1-v2
Metrics: normalized cumulative return average return per environment
Baselines: TD3+BC AWAC IQL TD3 Decision Transformer CQL XQL Diffusion Q-learning
Builds on: TD3 TD3+BC AWAC IQL D4RL
Closest papers by meaning
- Efficient Online Reinforcement Learning with Offline Data (ICML 2023 · similarity 0.93)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
- The Generalization Gap in Offline Reinforcement Learning (ICLR 2024 · similarity 0.91)
- Hybrid Reinforcement Learning from Offline Observation Alone (ICML 2024 · similarity 0.91)
- Horizon Reduction Makes RL Scalable (NeurIPS 2025 · similarity 0.91)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.