Research map / Reinforcement learning / Offline RL
Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SQOG, which improves offline RL Q-value estimation by smoothing OOD Q-targets within the convex hull and its neighborhood (CHN) using neighboring in-sample Q-values, achieving SOTA on D4RL with low compute.
Problem
Offline RL suffers from Q-value overestimation for out-of-distribution actions, and existing constraint methods become over-conservative in OOD regions, underestimating Q-values and hurting policy evaluation and improvement.
Approach
Defines a convex hull and its neighborhood (CHN) of dataset state-action pairs as a safe region for Q generalization, and introduces a Smooth Bellman Operator that sets OOD Q-values in the CHN to those of nearby in-sample actions; practically implemented by adding noise to dataset actions and regressing Q(s, a+noise) toward Q(s, a) on top of TD3+BC.
Key results
- SQOG achieves the highest Mujoco average (70.7) among compared methods, vs. MCQ 68.4 and TD3+BC 58.2 on D4RL.
- On Maze2d, SQOG averages 124.7 vs. MCQ 102.2; on Adroit total SQOG gets 149.6 vs. MCQ 123.3.
- SQOG runs in 0.4 h versus 8.0 h for MCQ on the 1M-step benchmark runs, roughly 20x faster.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SQOG instead of MCQ | D4RL Mujoco tasks · Mujoco Average | normalized average score: 68.4 -> 70.7 | Helped |
| Use SQOG instead of MCQ | D4RL Maze2d tasks · Maze2d Average | normalized average score: 102.2 -> 124.7 | Helped |
| Use SQOG instead of MCQ | D4RL Adroit tasks · Adroit Total | normalized total score: 123.3 -> 149.6 | Helped |
| Use SQOG instead of MCQ | Mujoco locomotion training · D4RL Mujoco | runtime (h): 8.0 -> 0.4 | Helped |
| Set α=300 instead of α=150 (ablation) | SQOG · Mujoco Average | normalized average score: 79.7 -> 61.4 | Hurt |
| Set α=200 instead of α=150 (ablation) | SQOG · Mujoco Average | normalized average score: 79.7 -> 68.2 | Hurt |
| Set α=100 instead of α=150 (ablation) | SQOG · Mujoco Average | normalized average score: 79.7 -> 70.8 | Hurt |
| Set α=50 instead of α=150 (ablation) | SQOG · Mujoco Average | normalized average score: 79.7 -> 65.8 | Hurt |
| Set α=200 instead of α=150 (ablation) | SQOG · halfcheetah-medium-v2 | normalized average score: 59.2±2.4 -> 57.7±1.6 | No clear effect |
| Set α=50 instead of α=150 (ablation) | SQOG · halfcheetah-medium-v2 | normalized average score: 59.2±2.4 -> 51.1±0.4 | Hurt |
| Set α=200 instead of α=150 (ablation) | SQOG · hopper-medium-v2 | normalized average score: 100.6±0.7 -> 91.6±6.0 | No clear effect |
| Use normal noise with clipping instead of uniform noise (ablation) | SQOG OOD action sampling · hopper-medium-v2, hopper-medium-replay-v2, halfcheetah-medium-v2 | normalized average score | Helped |
Surprising findings
- Hopper-medium-v2 result is ~70% above TD3+BC, while SQOG underperforms MCQ and DOGE on several datasets such as hopper-random and walker2d-medium.
Limitations
- Theoretical guarantees rely on NTK regime with infinite-width two-layer ReLU networks and on assumptions like OOD actions staying within the CHN.
- The method does not address OOD actions far outside the CHN.
- Performance is mixed: SQOG is below MCQ on hopper-random and halfcheetah-medium-dense-type settings and on several Adroit datasets.
Experimental setup
Datasets: D4RL halfcheetah D4RL hopper D4RL walker2d D4RL maze2d D4RL adroit Inverted Double Pendulum dataset from SAC policy
Benchmarks: D4RL
Metrics: normalized average score run time (h)
Baselines: BC TD3+BC CQL IQL DOGE MCQ BCQ
Builds on: TD3+BC DOGE MCQ
Closest papers by meaning
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.90)
- Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- On the Complexity of Offline Reinforcement Learning with Q*-Approximation and Partial Coverage (NeurIPS 2026 · similarity 0.89)
- Generative OOD-regularized Model-based Policy Optimization (NeurIPS 2026 · similarity 0.89)
- Doubly Mild Generalization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.