Research map / Reinforcement learning / Visual and representation RL
ODRL: A Benchmark for Off-Dynamics Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ODRL, the first benchmark for off-dynamics RL, with four online/offline source-target settings, locomotion, navigation and dexterous manipulation tasks with dynamics shifts, and a unified single-file implementation of many baselines.
Problem
Off-dynamics RL (transferring policies across domains with dynamics mismatch) lacks a standard, unified benchmark, so reported method gains are hard to compare.
Approach
A unified benchmark that crosses four online/offline source-target settings with diverse MuJoCo, AntMaze and Adroit tasks carrying friction, gravity, kinematic and morphology shifts, with off-dynamics baselines implemented in single files.
Key results
- No single off-dynamics RL method dominates across dynamics shifts; PAR is best on locomotion but fails on AntMaze and Adroit.
- AntMaze tasks are extremely hard: no algorithm achieves meaningful returns.
- On Adroit, only DARC and SAC_IW achieve comparatively good performance; off-dynamics methods often fail on dexterous manipulation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PAR for dynamics-aware adaptation | PAR · Locomotion tasks, Online-Online | normalized score | Helped |
| Use PAR for dynamics-aware adaptation | PAR · AntMaze and Adroit tasks, Online-Online | normalized score | Hurt |
| Adapt policies across maze barriers with state-based methods | ODRL methods · AntMaze tasks, Online-Online | return | Hurt |
| Use off-dynamics RL methods for dexterous manipulation | ODRL methods · Adroit tasks, Online-Online | normalized score | Hurt |
| Use DARC or SAC_IW for dexterous manipulation | DARC and SAC_IW · Adroit tasks, Online-Online | normalized score | Helped |
| Train SAC on data from both source and target domains | SAC · Numerous dynamics shifts, Online-Online | normalized score | Helped |
| Finetune a source-trained SAC policy directly on target data | SAC_tune · Morphology and gravity shifts, Online-Online | normalized score | Hurt |
| Use a higher-quality source-domain dataset | Offline-Online methods · Ant gravity shift 5.0 and Walker2d friction shift 0.5 | normalized score | Mixed |
| Treat the source and target domains as one mixed domain | Offline-Online baselines · Some offline-source tasks | normalized score | Helped |
| Use conservative value penalties instead of a BC term | CQL_SAC versus BC_PAR · Tasks with expert source-domain datasets | normalized score | Helped |
| Use an expert-quality target-domain dataset | Online-Offline methods · Ant kinematic ankle-joint hard shift and Walker2d morphology leg medium shift | normalized score | Mixed |
| Use a random-quality target-domain dataset | Online-Offline methods · Ant kinematic ankle-joint hard shift and Walker2d morphology leg medium shift | normalized score | Hurt |
Surprising findings
- Simply training SAC on mixed source and target data performs well on many dynamics shifts.
- A higher-quality source dataset does not necessarily yield better target performance, even with expert source data.
Limitations
- Primarily supports sim-to-sim adaptation; real-world combined dynamics shifts are not covered.
- Evaluation on a subset of tasks due to compute cost; many tasks not run.
- Only single-task settings with one source and one target domain.
Experimental setup
Datasets: D4RL MuJoCo offline datasets Adroit AntMaze MuJoCo
Benchmarks: ODRL
Metrics: normalized score episodic return
Baselines: DARC VGDF PAR H2O SAC SAC_IW SAC_tune RLPD IQL TD3_BC
Builds on: DARC H2O PAR VGDF CORL
Closest papers by meaning
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.92)
- Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.91)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.90)
- Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol (NeurIPS 2025 · similarity 0.90)
- Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.