Research map / Reinforcement learning / Offline RL

Generative OOD-regularized Model-based Policy Optimization

NeurIPS 2026 · Reinforcement learning › Offline RL · approach: Offline RL sample complexity theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GORMPO, which penalizes model-based offline RL rollout rewards using generative density estimators (VAE, RealNVP, DDPM, NeuralODE, KDE) to avoid OOD transitions, and evaluates OOD detection vs. policy performance on a medical MCS weaning dataset and sparse D4RL.

Problem

Offline RL policies and learned dynamics model rollouts drift into out-of-distribution state-action regions when the dataset is sparse, which is unsafe in high-stakes settings such as clinical decision-making.

Approach

Add a tanh-bounded density penalty, computed from a pretrained generative density estimator on the predicted next state and action, to the rolled-out reward in MBPO/MOBILE, and systematically compare five density estimator families on both OOD detection and downstream policy performance.

Key results

• On the proprietary medical dataset, GORMPO-RealNVP achieves reward 0.739 and WS 0.222, outperforming baselines; the paper reports a 17% reward gain over state-of-the-art baselines.
• MBPO-based GORMPO-VAE improves halfcheetah-medium-expert-sparse from 63.7 to 89.9, a 41% gain over MBPO.
• MOBILE-based GORMPO-RealNVP reaches 118.0 on walker2d-medium-expert-sparse versus 116.0 for MOBILE; the paper reports an 18.7% hopper improvement with MOBILE base.

Surprising findings

• GORMPO-DDPM achieves high reward despite weak OOD detection, attributed to penalty saturation producing aggressive conservatism.
• Diffusion-based density estimates have very high test NLL (e.g., 355134 on hopper), because a Gaussian approximation via Monte Carlo is used.

Limitations

• Relies on a density penalization threshold tau that can shift with distribution skewness and alter conservatism.
• The tanh penalty saturates in highly OOD regions, which may be too optimistic in safety-critical tasks.
• Main medical evaluation uses a proprietary dataset that is not released; the assumptions on density and model error are not verifiable.

Experimental setup

Datasets: proprietary cardiogenic shock MCS dataset D4RL halfcheetah-medium-expert (sparsified) D4RL hopper-medium-expert (sparsified) D4RL walker2d-medium-expert (sparsified)

Benchmarks: proprietary medical dataset (Transformer Digital Twin rollouts) D4RL halfcheetah, hopper, walker2d medium-expert sparse D4RL medium and medium-replay

Metrics: reward weaning score action change penalty ROC AUC accuracy TPR TNR normalized return

Baselines: SPOT MOBILE MBPO KDE-regularized MBPO

Builds on: MBPO MOBILE SPOT Neural ODE RealNVP DDPM

Closest papers by meaning

• Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
• Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
• Optimistic Model Rollouts for Pessimistic Offline Policy Optimization (AAAI 2024 · similarity 0.89)
• Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.89)
• Improving Generalization in Offline Reinforcement Learning via Adversarial Data Splitting (ICML 2024 · similarity 0.89)
• Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood (ICLR 2025 · similarity 0.89)