Research map / Reinforcement learning / Offline RL
Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MOREC, which learns a generalizable "dynamics reward" via adversarial IRL (GAIL-style discriminator ensemble) and uses it to select high-fidelity next-state samples and terminate rollouts in offline model-based RL, plugged into MOPO and MOBILE.
Problem
Learned dynamics models in offline model-based RL generalize poorly to unseen or out-of-distribution state-action transitions, causing compounding model errors during rollouts.
Approach
Treat the dynamics model as an agent maximizing a learned dynamics reward (via IRL over offline transitions, using an ensemble of discriminators), and at rollout time sample several candidate next states from the model ensemble and choose one via softmax over dynamics reward, terminating rollouts when reward falls below a threshold.
Key results
- On 12 D4RL tasks, MOREC-MOBILE and MOREC-MOPO reach average normalized returns of 82.8 and 83.7 (vs. 80.0 for MOBILE) and each solve 6/12 tasks.
- On 9 NeoRL tasks, MOREC-MOBILE averages 76.4 vs. 60.7 for MOBILE (about 25.9% improvement) and is the first method to solve 3/9 tasks.
- Dynamics reward correlates with model-transition MAE at Pearson -0.761 on walker2d-medium-replay rollouts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MOREC reward-consistent transition filtering to MOBILE | MOBILE · D4RL 12 tasks average | normalized average return: 80.0 -> 82.8 | Helped |
| Add MOREC reward-consistent transition filtering to MOPO | MOPO · D4RL 12 tasks average | normalized average return: 70.3 -> 83.7 | Helped |
| Add MOREC reward-consistent transition filtering to MOBILE | MOBILE · NeoRL 9 tasks average | normalized average return: 60.7 -> 76.4 | Helped |
| Add MOREC reward-consistent transition filtering to MOPO | MOPO · NeoRL 9 tasks average | normalized average return: 28.5 -> 70.3 | Helped |
| Add dynamics-reward transition filtering (ablation) | PPO policy learning in the refrigerator task · Refrigerator temperature-control task | return: -640 -> -78 | Helped |
| Add dynamics-reward transition filtering to model rollouts (ablation) | MOREC-MOPO · hfctah tasks | model transition MAE across rollout steps | Helped |
Surprising findings
- The learned dynamics reward assigns higher values to true transitions than to model transitions except in regions far from the offline data, and it can recover some distant unseen transitions on the synthetic task.
Limitations
- Performance depends on the accuracy of the dynamics reward recovered by IRL.
- Results are sensitive to per-task hyperparameters (beta, kappa), with MOBILE variant more sensitive to beta.
Experimental setup
Datasets: D4RL locomotion datasets NeoRL locomotion datasets synthetic refrigerator temperature-control data
Benchmarks: D4RL NeoRL synthetic refrigerator temperature-control task
Metrics: normalized average return solved tasks (score >= 95) Pearson correlation between dynamics reward and MAE model rollout MAE
Baselines: CQL TD3+BC EDAC MOPO COMBO TT RAMBO MOBILE MORel BC
Builds on: MOPO MOBILE GAIL maximum margin IRL
Closest papers by meaning
- Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning (ICML 2025 · similarity 0.91)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
- Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models (NeurIPS 2025 · similarity 0.90)
- Reasoning with Latent Diffusion in Offline Reinforcement Learning (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.