Research map / Reinforcement learning / Offline RL
The Generalization Gap in Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks offline learning (BC, BCQ, CQL, IQL, BCT, DT) on Procgen levels and WebShop instructions, finding offline methods generalize worse than online PPO and BC often beats offline RL and sequence models on unseen environments.
Problem
Generalization of offline learning algorithms to new environments (unseen levels or instructions) is underexplored because offline datasets typically come from a single environment instance.
Approach
Builds offline datasets from multiple contextual environment instances (Procgen levels, WebShop instructions) with train/validation/test splits over contexts, and evaluates offline RL, sequence modeling, and BC against online PPO on unseen contexts, also studying the effect of data diversity and size.
Key results
- On Procgen 1M expert data, BC achieves higher test performance than offline RL and sequence-modeling baselines, but all offline methods underperform online PPO on unseen levels.
- Increasing the number of training levels at fixed 1M transitions improves test performance for all offline algorithms, while increasing dataset size at fixed 200 levels yields only slight gains and a roughly constant generalization gap.
- On WebShop human demonstrations, BC achieves higher score than BCQ and CQL on test instructions, though BCQ has a higher success rate during training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use behavioral cloning instead of offline RL or sequence modeling | Procgen agents trained on 1M expert demonstrations · Procgen unseen test levels | mean normalized return | Helped |
| Use behavioral cloning instead of offline RL or sequence modeling | Procgen agents trained on mixed expert-suboptimal data · Procgen unseen test levels | return | Helped |
| Train with data from more levels while keeping dataset size fixed | Procgen offline learning algorithms · Procgen unseen levels | generalization performance | Helped |
| Increase transition count while keeping training-level diversity fixed | Procgen offline learning algorithms · Procgen unseen levels | test performance | No clear effect |
| Use online PPO rather than offline learning | Procgen agents trained from expert data · Procgen unseen environments | mean return | Helped |
| Use BC rather than CQL or BCQ | WebShop agents trained on human demonstrations · WebShop test instructions | average score | Helped |
| Use BC rather than BCQ or CQL | WebShop agents trained on suboptimal demonstrations · WebShop | score | Helped |
| Train on at least 500 episodes instead of smaller WebShop datasets | BC, BCQ, and CQL · WebShop suboptimal demonstrations | train and test performance | Helped |
| Increase WebShop training duration | BC, CQL, and BCQ trained on human demonstrations · WebShop test instructions | test performance | Hurt |
| Use offline RL rather than BC on single-environment suboptimal demonstrations | Procgen agents · Same-level evaluation, most games | return | Mixed |
Surprising findings
- Behavioral cloning outperforms offline RL methods on unseen levels even when trained on mixed expert-suboptimal data, contrary to prior singleton-environment results.
- Increasing dataset size without increasing the number of training levels does not improve generalization for any offline algorithm.
Limitations
- Only discrete action spaces are studied.
- WebShop experiments are limited to BC, BCQ and CQL because transformer policies cannot handle its long state contexts, and only the choice task is used.
- Procgen main datasets use 1M transitions, smaller than the 25M used by PPO training.
Experimental setup
Datasets: Procgen expert data (1M, 10M transitions) Procgen mixed expert-suboptimal data Procgen suboptimal data (25M transitions) WebShop human demonstrations (452 trajectories) WebShop IL suboptimal trajectories
Benchmarks: Procgen WebShop
Metrics: IQM of min-max normalized return mean return success rate generalization gap
Baselines: PPO BC BCQ CQL IQL BCT DT
Builds on: Procgen WebShop Decision Transformer Implicit Q-Learning Conservative Q-Learning
Closest papers by meaning
- Doubly Mild Generalization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Simple Ingredients for Offline Reinforcement Learning (ICML 2024 · similarity 0.91)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.91)
- Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining (ICLR 2025 · similarity 0.90)
- Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.