Research map / Reinforcement learning / Offline RL
Any-step Dynamics Model Improves Future Predictions for Online and Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Any-step Dynamics Model (ADM), an RNN-based model that predicts next states from variable-length action plans to reduce compounding error, and plugs it into MBPO-style online (ADMPO-ON) and MOPO-style offline (ADMPO-OFF) algorithms.
Problem
Compounding error in model roll-outs due to bootstrapped one-step prediction hurts both sample efficiency in online model-based RL and reliable uncertainty estimation in offline model-based RL.
Approach
Replace one-step bootstrapped dynamics with a single RNN-based model trained to predict the next state from any backtracked k-step plan (k in [1,m]); random backtracking during roll-outs reduces bootstrapping, and the disagreement across backtracking lengths serves as an ensemble-free uncertainty estimate used as a penalty offline.
Key results
- ADMPO-OFF achieves an average normalized score of 81.0 on 12 D4RL MuJoCo v2 datasets, above the next best MOBILE (80.0) and CBOP (79.3).
- ADMPO-OFF averages 64.0 on NeoRL nine tasks, versus 60.7 for MOBILE and 56.1 for CQL.
- ADMPO-ON reaches about 100% of SAC asymptotic performance on Humanoid after 150k steps, while DDPPO needs about 200k steps (about 1.33x faster).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ADM instead of an ensemble dynamics model | Offline model roll-outs on D4RL MuJoCo · D4RL MuJoCo average | normalized score: 80.0 -> 81.0 | Helped |
| Use ADM instead of an ensemble dynamics model | Offline model roll-outs on NeoRL MuJoCo · NeoRL MuJoCo average | normalized score: 60.7 -> 64.0 | Helped |
| Replace MBPO's ensemble dynamics model with ADM | Online model-based policy optimization · Humanoid-v3 | sample efficiency: about 1.33x faster than DDPPO | Helped |
| Use variable-length backtracking for future-state prediction | ADM roll-outs on D4RL hopper-medium-v2, hopper-medium-replay-v2, walker2d-medium-v2, and walker2d-medium-replay-v2 · D4RL dynamics model evaluation | compounding error | Helped |
| Vary the maximum backtracking length (ablation) | ADMPO-OFF · hopper-medium-v2 and walker2d-medium-v2 | performance | No clear effect |
| Use ADM uncertainty instead of ensemble uncertainty | Offline uncertainty quantification · hopper-medium-replay-v2 and walker2d-medium-replay-v2 | correlation coefficient between model uncertainty and model error: 0.98 | Helped |
Surprising findings
- ADM's compounding error curves stay near zero while the ensemble and bootstrapping RNN models grow exponentially beyond a roll-out length threshold.
- ADMPO-OFF's performance is reported as not very sensitive to the maximum backtracking length m.
Experimental setup
Datasets: D4RL MuJoCo v2 datasets NeoRL MuJoCo datasets MuJoCo-v3 online interaction
Benchmarks: MuJoCo Hopper MuJoCo Walker2d MuJoCo Ant MuJoCo Humanoid D4RL MuJoCo NeoRL
Metrics: normalized score sample efficiency (learning curves) compounding error model error correlation with uncertainty
Baselines: SAC STEVE MBPO BMPO DDPPO BC CQL TD3+BC EDAC MOPO COMBO RAMBO
Builds on: MBPO MOPO PEVI SAC
Closest papers by meaning
- ADM-v2: Pursuing Full-Horizon Roll-out in Dynamics Models for Offline Policy Learning and Evaluation (ICLR 2026 · similarity 0.93)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Optimistic Model Rollouts for Pessimistic Offline Policy Optimization (AAAI 2024 · similarity 0.88)
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.88)
- Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.