Research map / Reinforcement learning / Offline RL
Optimistic Model Rollouts for Pessimistic Offline Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ORPO, which trains an optimistic rollout policy in an optimistic MDP to generate OOD model rollouts, relabels them with pessimistic penalized rewards, and optimizes the output policy in the pessimistic MDP for model-based offline RL.
Problem
Model-based offline RL with pessimistic MDPs (P-MDP) penalizes out-of-distribution regions, which under-utilizes the generalization ability of learned dynamics models.
Approach
Decouple rollout generation from pessimistic optimization: an optimistic rollout policy (trained in an O-MDP with flipped uncertainty bonus) samples more OOD model rollouts; these are relabeled with pessimistic penalized rewards and used to train the output policy in the P-MDP.
Key results
- ORPO raises MOPO's total average normalized score on 12 D4RL v2 datasets from 610.0 to 809.9 (over 30%), improving 11 of 12 datasets.
- ORPO is competitive or best on 8 of 12 D4RL datasets, e.g., HalfCheetah-medium-expert 101.5 vs MOPO 80.8 and CABI 105.0.
- On Halfcheetah-jump-hard, ORPO scores 4867.9 versus MOPO's -1881.8 and COMBO's 2782.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ORPO instead of MOPO on D4RL | Model-based offline RL · D4RL v2 datasets (12 datasets) | total average normalized score: 610.0 to 809.9; more than 30% | Helped |
| Use ORPO instead of MOPO | HalfCheetah-jump · HalfCheetah-jump | average return: 4411.8±642.9 -> 5218.0±128.5 | Helped |
| Use ORPO instead of MOPO | HalfCheetah-jump-hard · HalfCheetah-jump-hard | average return: -1881.8±1342.2 -> 4867.9±381.6 | Helped |
| Use optimistic rollouts with pessimistic optimization instead of P-MDP-only rollouts | RiskWorld toy task · RiskWorld toy task | average cumulative reward: 7.3 -> 35.3 | Helped |
| Use O-MDP instead of P-MDP for rollout and policy optimization (OROO) (ablation) | OROO · reported offline RL dataset | real-environment return | No clear effect |
| Use ORPO instead of OROO with O-MDP-only optimization (ablation) | ORPO · reported offline RL dataset | real-environment return | Helped |
| Vary λᵒ across -10, -1, -0.1, -0.01, 0.01, 0.1, 1, and 10 (ablation) | ORPO · HalfCheetah-Random-v2 | normalized score: 27.7; 39.1; 40.6; 38.5; 40.2; 41.2; 41.6; 26.0 | Mixed |
| Vary λᵒ across -10, -1, -0.1, -0.01, 0.01, 0.1, 1, and 10 (ablation) | ORPO · Walker2d-Medium-Replay-v2 | normalized score: 43.7; 85.5; 91.1; 91.3; 91.8; 88.6; 8.6; 0.1 | Mixed |
| Replace SAC with TD3+BC for MOPO policy optimization (ablation) | MOPO · D4RL benchmarks | performance | No clear effect |
| Use SAC to optimize both rollout and output policies instead of offline RL optimization for the output policy (ablation) | ORPO (SAC) · D4RL benchmarks | performance | Hurt |
| Remove pessimism by replacing ORPO's P-MDP with the model MDP (ablation) | ORPO · D4RL benchmarks | performance | Hurt |
| Use O-MDP instead of P-MDP for policy optimization (ablation) | OROO · reported model-rollout comparison | model-MDP return | Helped |
Surprising findings
- Optimistic rollouts alone (OROO, no pessimism in the output policy) can outperform MBPO and sometimes MOPO, yet degrade real-environment return due to larger model error.
- ORPO without pessimism underperforms, and ORPO is robust to lambda^o across three orders of magnitude.
Limitations
- Additional training time overhead for the optimistic rollout policy.
- In tasks where OOD actions are strictly prohibited, ORPO can hurt relative to P-MDP baselines.
- Optimism increases model error when optimizing policies directly in the O-MDP.
Experimental setup
Datasets: D4RL halfcheetah v2 D4RL hopper v2 D4RL walker2d v2 Halfcheetah-jump (Yu et al.) Halfcheetah-jump-hard (constructed)
Benchmarks: D4RL Halfcheetah-jump Halfcheetah-jump-hard
Metrics: normalized score average return
Baselines: CQL TD3+BC IQL PBRL MOPO MOReL COMBO RAMBO CABI
Builds on: MOPO MBPO TD3+BC SAC D4RL
Closest papers by meaning
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
- Provable Zero-Shot Generalization in Offline Reinforcement Learning (ICML 2025 · similarity 0.90)
- Model-based Reinforcement Learning for Confounded POMDPs (ICML 2024 · similarity 0.90)
- Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol (NeurIPS 2025 · similarity 0.90)
- Generative OOD-regularized Model-based Policy Optimization (NeurIPS 2026 · similarity 0.89)
- A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.