Research map / Reinforcement learning / Offline RL
Dynamic Uncertainty Estimation for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DURL, an offline RL method that combines base density-truncated OOD sampling (using Flow-GAN density estimates) with learnable dynamic uncertainty penalties on Bootstrapped Q-ensembles to regularize OOD actions without excessive pessimism.
Problem
Offline RL suffers from distributional shift and extrapolation error; existing conservative value methods often make Q-values of OOD actions overly pessimistic, hurting Q-function generalization and policy improvement.
Approach
Truncate OOD-sampled actions by Flow-GAN density (discarding the highest-density N samples so only low-density OOD data is penalized), and learn the uncertainty penalty coefficients (beta, omega) dynamically by minimizing the target-estimate error, instead of fixing them.
Key results
- DURL achieves 84.5 average normalized score on D4RL MuJoCo v2 vs 78.8 for the best baseline MCQ and 75.1 for CSVE.
- Applying DT to CQL raises its MuJoCo average from 66.7 to 75.9 (+13.8%).
- On Adroit, DURL averages 70.9 vs 66.7 for CSVE and 65.1 for PBRL, outperforming baselines on six of eight tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DURL instead of MCQ | D4RL MuJoCo -v2 · D4RL MuJoCo -v2 average | normalized score: 78.8 ± 1.3 -> 84.5 ± 1.3 | Helped |
| Use DURL instead of CSVE | Adroit · Adroit average | normalized score: 66.7 ± 2.6 -> 70.9 ± 3.9 | Helped |
| Add DT | CQL · D4RL MuJoCo -v2 average | normalized score: 66.7 -> 75.9 (+13.8%) | Helped |
| Add DT | PBRL · D4RL MuJoCo -v2 average | normalized score: 75.0 -> 79.2 (+5.6%) | Helped |
| Add DT | MCQ · D4RL MuJoCo -v2 average | normalized score: 78.8 -> 83.0 (+5.3%) | Helped |
| Add DT to MCQ | MCQ · Hopper-random | normalized score: 31.8 -> 30.5 (-4.1%) | Hurt |
| Add dynamic uncertainty estimation (ablation) | DU-N+DT · Walker2d-medium-v2 | normalized score | Helped |
| Add DT (ablation) | DU-N+DE · Walker2d-medium-v2 | normalized score | Helped |
| Add dynamic uncertainty estimation (ablation) | DU-N · Walker2d-medium-v2 | normalized score | Helped |
| Set initial ω to 5 (ablation) | DURL · Walker2d-medium-v2 | normalized score | Helped |
| Set initial ω to 3 (ablation) | DURL · Walker2d-medium-v2 | normalized score | Hurt |
| Vary β from 0.0001 to 0.1 (ablation) | DURL · Walker2d-medium-v2 | normalized score | No clear effect |
Surprising findings
- Applying the density-truncated OOD sampling (DT) to CQL, PBRL, and MCQ improved them, though MCQ+DT dropped on Hopper-random.
Limitations
- Relies on many hyperparameters (initial omega, beta, density truncation N) with sensitivity shown in ablations.
- Method is limited to model-free offline RL; extension to model-based settings left to future work.
- Baseline results for most methods were obtained by re-running official code, and CSVE numbers are taken from its paper.
Experimental setup
Datasets: D4RL MuJoCo v2 D4RL Adroit D4RL Maze2d
Benchmarks: D4RL MuJoCo D4RL Adroit D4RL Maze2d
Metrics: normalized average score Q-value absolute bias
Baselines: CQL UWAC TD3-BC MCQ PBRL CSVE EDAC IQL
Builds on: Bootstrapped DQN Flow-GAN PBRL MCQ SAC-N
Closest papers by meaning
- Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model (NeurIPS 2024 · similarity 0.92)
- Selective Uncertainty Propagation in Offline RL (AAAI 2025 · similarity 0.91)
- A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch (NeurIPS 2024 · similarity 0.91)
- SUMO: Search-Based Uncertainty Estimation for Model-Based Offline Reinforcement Learning (AAAI 2025 · similarity 0.91)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.91)
- Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.