Research map / Reinforcement learning / Offline RL
Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCAS, an offline RL method that unifies value-aware OOD state correction (aligning policy-induced dynamics on perturbed states with a value-skewed next-state distribution) with OOD action suppression, using a single hyperparameter set across D4RL and NeoRL.
Problem
Offline RL agents entering out-of-distribution states at test time behave uncontrollably, and prior OOD state correction methods are separate from OOD action suppression, need complex distribution modeling, and treat all in-distribution states equally, including low-value ones.
Approach
Derive a closed-form value-aware next-state distribution (exp(alpha V) reweighted dataset transitions), align policy-induced dynamics on Gaussian-perturbed states with it via a KL-derived regularizer using a learned deterministic dynamics model, and combine with TD3+BC-style Q maximization; the regularizer is shown to also suppress OOD actions.
Key results
- On D4RL Gym locomotion v2 tasks SCAS achieves a total normalized score of 810.1, the highest among compared methods, with one hyperparameter set and no per-dataset tuning.
- On AntMaze, SCAS reaches 410.8 total normalized score versus 392.4 for IQL and 289.2 for SDC.
- In perturbed test environments SCAS degrades less than TD3+BC, CQL and SDC as the number of noisy action steps increases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SCAS regularization to ordinary off-policy RL (ablation) | SCAS · D4RL | learned Q-value stability | Helped |
| Remove SCAS regularization (λ=0) (ablation) | SCAS · D4RL | learned Q-value stability | Hurt |
| Set inverse temperature α=0, removing value-aware weighting (ablation) | SCAS · D4RL parameter study | performance | Hurt |
| Set inverse temperature α=10 (ablation) | SCAS · D4RL parameter study | performance | Hurt |
| Vary balance coefficient λ across [0,1] (ablation) | SCAS · D4RL parameter study | performance | No clear effect |
| Set λ=1, removing the RL improvement (max Q) signal (ablation) | SCAS · D4RL parameter study | performance | No clear effect |
| Add value-aware SCAS regularization rather than vanilla OOD state correction (ablation) | SCAS · D4RL | performance | Helped |
| Use SCAS in place of prior methods | SCAS · Perturbed D4RL environments | normalized score | Helped |
Surprising findings
- Even with lambda=1 (removing the max-Q term), SCAS still performs well on most tasks, suggesting the value-aware state correction term alone provides policy improvement.
Limitations
- Focuses on continuous control; discrete settings would need different state perturbation strategies.
- Relies on a learned deterministic one-step dynamics model whose accuracy affects performance; more advanced dynamics models (ensembles, diffusion) are left for future work.
- Very large alpha (alpha=10) degrades performance, indicating sensitivity to the inverse temperature.
Experimental setup
Datasets: D4RL NeoRL
Benchmarks: D4RL Gym locomotion D4RL AntMaze NeoRL
Metrics: normalized score Q-value divergence robustness under perturbation steps
Baselines: BC MOPO OneStep RL CQL TD3+BC IQL SDC OSR
Builds on: TD3+BC SDC OSR CQL
Closest papers by meaning
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.92)
- ODRL: A Benchmark for Off-Dynamics Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Generative OOD-regularized Model-based Policy Optimization (NeurIPS 2026 · similarity 0.89)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimation (AAAI 2025 · similarity 0.89)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.