Research map / Reinforcement learning / Offline RL
OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OCEAN, a plug-in uncertainty-gated exploratory rollout method that adds conservative exploration to model-based offline RL (MOPO, COMBO, RAMBO) and improves D4RL MuJoCo scores.
Problem
Model-based offline RL rollouts rely on maximum-entropy or uncertainty-stopped policy rollouts that explore the learned model poorly (biased exploration), limiting performance and stability, while exploration in learned models risks large model errors.
Approach
During model rollouts, add Gaussian-noise exploration only from states whose sampled exploration actions have low ensemble uncertainty, pick conservative (min/median/random-uncertainty) exploratory transitions, and truncate rollouts with an automatic uncertainty threshold (lambda times the dataset max uncertainty).
Key results
- MOPO+OCEAN averages 92.5 normalized score across nine D4RL MuJoCo tasks vs 77.4 for MOPO, a 19.5% gain with 54.2% lower std.
- COMBO+OCEAN averages 92.3 vs 89.8 for COMBO; RAMBO+OCEAN averages 89.0 vs 84.1 for RAMBO.
- On hopper tasks MOPO+OCEAN improves from 70.7 to 100.3 average score.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add OCEAN rollout to MOPO | MOPO · D4RL MuJoCo v2, average score | normalized average score: 77.4(5.9) -> 92.5(2.7) | Helped |
| Add OCEAN rollout to COMBO | COMBO · D4RL MuJoCo v2, average score | normalized average score: 89.8(2.4) -> 92.3(2.0) | Helped |
| Add OCEAN rollout to RAMBO | RAMBO · D4RL MuJoCo v2, average score | normalized average score: 84.1(6.1) -> 89.0(4.5) | Helped |
| Add Gaussian action noise without conservative constraints (ablation) | MOPO on walker2d-medium-replay-v2 · walker2d-medium-replay-v2 | score | Hurt |
| Add conservative uncertainty constraints to noisy rollout (ablation) | MOPO on walker2d-medium-replay-v2 · walker2d-medium-replay-v2 | score | Helped |
| Add exploration while retaining trajectory truncation (ablation) | MOPO on walker2d-medium-replay-v2 · walker2d-medium-replay-v2 | score | Helped |
| Use Max Aleatoric uncertainty instead of Max Pairwise Diff (ablation) | RAMBO+OCEAN · walker2d-medium-replay-v2 | score | Helped |
| Use Ensemble Standard Variance instead of Max Pairwise Diff (ablation) | RAMBO+OCEAN · walker2d-medium-replay-v2 | score | Helped |
| Select the exploration action with minimum uncertainty rather than median or random uncertainty (ablation) | RAMBO+OCEAN · walker2d-medium-replay-v2 | score | Helped |
| Vary the penalty threshold uT (ablation) | MOPO+OCEAN, COMBO+OCEAN, and RAMBO+OCEAN · walker2d-medium-replay-v2 | score | Mixed |
| Use batch average uncertainty as the penalty threshold (ablation) | MOPO+OCEAN, COMBO+OCEAN, and RAMBO+OCEAN · walker2d-medium-replay-v2 | score | Helped |
| Vary Gaussian noise standard deviation δ (ablation) | MOPO+OCEAN, COMBO+OCEAN, and RAMBO+OCEAN · walker2d-medium-replay-v2 | score | Mixed |
Surprising findings
- Unconstrained noise exploration (MOPO with action noise) performs similarly to original MOPO in later training, showing that exploration without conservatism does not help.
- Trajectory truncation alone (without exploration) lowers performance by 14.3 points versus full OCEAN on walker2d-medium-replay, indicating exploration itself matters.
Limitations
- Gains for COMBO and RAMBO are limited, attributed to their existing conservatism or implicit exploration.
- Performance is sensitive to the penalty threshold u_T and noise std delta, requiring tuning per task.
- Evaluated only on D4RL MuJoCo v2 tasks (halfcheetah, hopper, walker2d).
Experimental setup
Datasets: D4RL halfcheetah-v2 datasets D4RL hopper-v2 datasets D4RL walker2d-v2 datasets
Benchmarks: D4RL MuJoCo v2
Metrics: normalized average score standard deviation
Baselines: MOPO COMBO RAMBO CQL TD3+BC IQL TATU
Builds on: MOPO COMBO RAMBO MBPO TATU
Closest papers by meaning
- MOPO: Model-based Offline Policy Optimization (NeurIPS 2020 · similarity 0.94)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning (AAAI 2025 · similarity 0.91)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Optimistic Model Rollouts for Pessimistic Offline Policy Optimization (AAAI 2024 · similarity 0.90)
- Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.