Research map / Reinforcement learning / Offline RL

Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning

NeurIPS 2024 · Reinforcement learning › Offline RL · approach: Pessimism and uncertainty in offline RL · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MOMBO, which replaces Monte Carlo sampling of the Bellman target in pessimistic value iteration with deterministic moment-matching propagation of next-state uncertainty through a ReLU Q-network, yielding faster convergence on D4RL.

Problem

Model-based offline RL with pessimistic value iteration relies on Monte Carlo sampling to estimate the Bellman target uncertainty penalty, and this sampling randomness slows convergence and yields loose suboptimality bounds.

Approach

Propagate the Gaussian next-state uncertainty deterministically through the critic by moment matching ReLU activations layer by layer (means and variances only), and use a lower confidence bound on the resulting Bellman target distribution as the penalty, requiring only two forward passes instead of N samples.

Key results

• Across 12 D4RL tasks, MOMBO average normalized reward is 79.5 vs 79.8 for MOBILE and 67.6 for MOPO, with AULC 70.5 vs 67.7 (MOBILE) and 57.5 (MOPO).
• MOMBO outperforms baselines in uncertainty tightness on all 12 tasks and in accuracy on 9 of 12 tasks.
• At 1% expert data in the mixed dataset, MOMBO achieves normalized reward 37.0 vs 32.4 (MOBILE) and 27.9 (MOPO) and AULC 29.5 vs 28.0 and 23.5.

Measured effects reported in the paper

Surprising findings

• Sampling-based Bellman target uncertainty does not shrink even after 10000 samples through the Q-network, while moment matching with two passes gives similar mean and smaller variance.
• On the hopper-random task MOMBO has high variance due to one failed repetition, and its normalized reward (25.4) is below MOPO and MOBILE.

Limitations

• Accuracy of learned environment models bounds performance; choice of confidence radius beta is decisive.
• Theory assumes normally distributed Q-values; the tighter bound is compared against a baseline bound the authors derived themselves.
• Method restricted to activations whose first two moments are tractable (ReLU used); BatchNorm and similar layers are not handled.

Experimental setup

Datasets: D4RL halfcheetah-v2 D4RL hopper-v2 D4RL walker2d-v2 mixed offline RL dataset

Benchmarks: D4RL halfcheetah D4RL hopper D4RL walker2d

Metrics: normalized reward area under learning curve (AULC) accuracy of uncertainty quantifier tightness of uncertainty quantifier

Baselines: MOPO MOBILE COMBO TT RAMBO

Builds on: MOPO MOBILE MBPO Soft Actor-Critic progressive moment matching

Closest papers by meaning

• Selective Uncertainty Propagation in Offline RL (AAAI 2025 · similarity 0.91)
• A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.90)
• Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
• Dynamic Uncertainty Estimation for Offline Reinforcement Learning (AAAI 2025 · similarity 0.89)
• Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
• Neural Network Approximation for Pessimistic Offline Reinforcement Learning (AAAI 2024 · similarity 0.89)