Research map / Reinforcement learning / Offline RL
OGBench: Benchmarking Offline Goal-Conditioned RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OGBench, a benchmark of 8 environment types, 85 datasets and reference implementations of 6 offline goal-conditioned RL algorithms, probing stitching, long-horizon reasoning, pixel inputs and stochasticity.
Problem
Lack of a standard, systematic benchmark for evaluating offline goal-conditioned RL algorithms, with existing tasks being either single-goal D4RL tasks, simple Fetch tasks, or bespoke tasks for individual abilities.
Approach
A unified suite of locomotion, manipulation and drawing environments with controllable data-generation scripts, multi-goal evaluation (five state-goal pairs per task), state and pixel observations, and well-tuned reference implementations of six offline GCRL methods.
Key results
- HIQL is strongest overall, e.g. 91% on antmaze-large-navigate and 89% on humanoidmaze-medium-navigate, while no method dominates all categories.
- Single-goal D4RL antmaze-large evaluation reverses the GCIQL vs QRL ranking seen under multi-goal evaluation on OGBench antmaze-large-navigate.
- Removing action noise in data collection drops cube-single-play performance from 99% to 6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HIQL rather than one-step CRL for goal stitching | offline GCRL on locomotion tasks · Visual locomotion stitch datasets | success rate | Helped |
| Use CRL rather than value-only HIQL or QRL in stochastic teleport mazes | offline GCRL · AntMaze teleport | success rate | Helped |
| Use hierarchical HIQL rather than non-hierarchical methods overall | offline GCRL benchmark tasks · Across benchmark categories | success rate | Mixed |
| Use CRL among non-hierarchical methods | offline GCRL · Locomotion tasks | success rate | Helped |
| Use GCIQL among non-hierarchical methods | offline GCRL · Manipulation tasks | success rate | Helped |
| Use GCIVL among benchmarked methods | offline GCRL · Powderworld drawing tasks | success rate | Helped |
| Add Gaussian action noise to expert data rather than collect noise-free expert actions (ablation) | GCIQL on Cube single noisy · Cube single | success rate: 6% -> 99% | Helped |
| Evaluate multiple goals rather than one fixed goal | offline GCRL evaluation on AntMaze large · AntMaze large navigate | success rate | Mixed |
| Use the OGBench reference implementation rather than previously reported implementations | CRL · D4RL AntMaze large diverse v2 | success rate: 54 -> 79 | Helped |
| Use the OGBench reference implementation rather than previously reported implementations | CRL · D4RL AntMaze large play v2 | success rate: 49 -> 74 | Helped |
Surprising findings
- Removing Gaussian action noise from expert data collection drops performance from 99% to 6% even on basic cube pick-and-place, showing coverage matters more than optimality.
- CRL, a one-step method fitting behavioral value, significantly outperforms GCIQL on locomotion tasks, contrary to the expectation that optimal Q-learning is better.
Limitations
- All tasks assume identical dynamics between training and evaluation environments.
- Does not test visual generalization to entirely new objects.
- Synthetic, controllable datasets and transparent arm sacrifice some realism.
Experimental setup
Datasets: OGBench datasets (85 datasets generated by scripted or SAC expert policies) D4RL antmaze-large-diverse-v2 D4RL antmaze-large-play-v2
Benchmarks: OGBench D4RL antmaze-large
Metrics: average binary success rate across five test-time goals
Baselines: GCBC GCIVL GCIQL QRL CRL HIQL
Builds on: D4RL implicit Q-learning contrastive RL hierarchical implicit Q-learning quasimetric RL
Closest papers by meaning
- Score Models for Offline Goal-Conditioned Reinforcement Learning (ICLR 2024 · similarity 0.92)
- Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.91)
- ODRL: A Benchmark for Off-Dynamics Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Conservative Offline Goal-Conditioned Implicit V-Learning (ICML 2025 · similarity 0.89)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.