Research map / Reinforcement learning / Offline RL
The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that offline model-based RL methods fail even with the true dynamics because truncated rollouts create edge-of-reach states whose values are never trained, and proposes RAVL, which uses ensemble-minimum value pessimism to fix this without dynamics penalties.
Problem
Offline model-based RL relies on short k-step model rollouts, and existing methods attribute failures to dynamics model error; the paper asks why methods fail when the learned model is replaced by the true dynamics.
Approach
Identifies the edge-of-reach problem: states reachable only at the final rollout step are used as Bellman targets but never updated, causing value overestimation. RAVL detects these via Q-ensemble variance and penalizes them by taking the minimum over an ensemble of Q-functions (EDAC-style diversity), replacing dynamics uncertainty penalties.
Key results
- With true dynamics, MOBILE (representing all dynamics-penalty methods) collapses on Hopper and Walker2d (e.g., Walker2d-medium 7.2 vs RAVL 101.2), while RAVL stays near-optimal across the same datasets.
- On D4RL with learned dynamics, RAVL largely matches SOTA (e.g., Halfcheetah-medium 78.7 vs MOBILE 74.6; Walker2d-medexp 115.5 vs MOBILE 115.2).
- On pixel-based V-D4RL, RAVL achieves overall 37.6 vs Offline DV2 35.5 and LOMPO 26.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace approximate dynamics with true dynamics in MOBILE | MOBILE · D4RL MuJoCo v2, Hopper-medium | normalized score: 106.6 -> 17.2 | Hurt |
| Replace approximate dynamics with true dynamics in MOBILE | MOBILE · D4RL MuJoCo v2, Walker2d-medium | normalized score: 87.7 -> 7.2 | Hurt |
| Use reach-aware value pessimism with true dynamics rather than MOBILE | D4RL MuJoCo v2 · Hopper-medium | normalized score: 17.2$\pm$11.1 -> 101.6$\pm$0.9 | Helped |
| Use reach-aware value pessimism with learned dynamics rather than the strongest listed baseline | RAVL · HalfCheetah-medium | normalized score: 77.9 -> 78.7$\pm$2.0 | Helped |
| Use reach-aware value pessimism with learned dynamics rather than the strongest listed baseline | RAVL · Walker2d-medium | normalized score: 92.5 -> 86.3$\pm$1.6 | Hurt |
| Use reach-aware value pessimism | RAVL · V-D4RL Overall | normalized score: 35.5$\pm$12.0 -> 37.6$\pm$7.0 | Helped |
| Set diversity regularizer eta to 10.0 instead of 1.0 (ablation) | RAVL · HalfCheetah-medium | normalized score: $78.7\pm 2.0$ -> $74.8\pm 1.5$ | Hurt |
| Set diversity regularizer eta to 100.0 instead of 1.0 (ablation) | RAVL · HalfCheetah-medium | normalized score: $78.7\pm 2.0$ -> $72.1\pm 2.2$ | Hurt |
| Set diversity regularizer eta to 10.0 instead of 1.0 (ablation) | RAVL · HalfCheetah-mixed | normalized score: $61.7\pm 5.6$ -> $73.8\pm 1.9$ | Helped |
| Set diversity regularizer eta to 100.0 instead of 1.0 (ablation) | RAVL · HalfCheetah-mixed | normalized score: $61.7\pm 5.6$ -> $74.9\pm 2.0$ | Helped |
| Increase Q-ensemble size from N=2 to N=100 (ablation) | RAVL · Runtime experiment | time relative to N=2: (baseline) -> $\times 1.01\pm0.02$ | No clear effect |
| Add uncertainty penalty with weight lambda=10 instead of lambda=0 (ablation) | RAVL · V-D4RL DeepMind Control Suite | performance | Helped |
Surprising findings
- All existing dynamics-penalized offline model-based methods fail when the learned model is replaced by the true dynamics, even though the learned model is already highly accurate.
- Per-step model rewards stay on the order of 10^0 and match true rewards, so model exploitation cannot explain Q-value explosion beyond 10^10.
Limitations
- Edge-of-reach effects may mix with dynamics error in environments where the model is poor; balancing the two is left to future work.
- Results are on MuJoCo/D4RL and V-D4RL benchmarks where models are highly accurate; generalization to other settings is untested.
- RAVL does not clearly outperform SOTA on several settings (e.g., Walker2d-random, Hopper-medium), and uses EDAC's diversity regularizer.
Experimental setup
Datasets: D4RL MuJoCo v2 V-D4RL
Benchmarks: D4RL MuJoCo v2 V-D4RL
Metrics: normalized return Q-value growth
Baselines: MOPO COMBO RAMBO MOBILE MOReL BC CQL EDAC LOMPO Offline DV2
Builds on: MOPO MBPO EDAC SAC DreamerV2
Closest papers by meaning
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- OCEAN-MBRL: Offline Conservative Exploration for Model-Based Offline Reinforcement Learning (AAAI 2024 · similarity 0.91)
- Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol (NeurIPS 2025 · similarity 0.90)
- Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.90)
- Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.