Research map / Reinforcement learning / Offline RL
Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes selective state-adaptive regularization that learns state-wise regularization coefficients for CQL and TD3+BC, restricted to high-quality sub-datasets, and uses annealed coefficients for offline-to-online fine-tuning on D4RL.
Problem
Offline RL uses a fixed global regularization strength that cannot balance trust in Bellman updates against imitation of dataset actions across tasks, training stages, and state data densities, which also hampers offline-to-online fine-tuning.
Approach
Replace the global regularization coefficient with a state-dependent coefficient network trained so that dataset action log-likelihood exceeds a distribution-aware threshold (a trust region from the policy's mean and std), apply the regularization selectively to high-return or IQL-advantage-filtered sub-datasets, and linearly anneal the coefficients online.
Key results
- Offline: CQL(SA) reaches 1139.1 locomotion total vs 1030.4 for CQL and 1116.7 for TD3+BC(SA) vs 1000.8 for TD3+BC on MuJoCo v2 datasets.
- Offline AntMaze totals rise from 213.0 (CQL base average) to 341.4 with the method averaged across both backbones; CQL(SA) reaches 406.8 vs 294.1 for CQL.
- Offline-to-online (250k steps): CQL(SA) locomotion total 1278.4 vs 1068.4 for CQL and 1057.4 for IQL; AntMaze total 494.3 vs 405.5 for CQL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add selective state-adaptive regularization | TD3+BC · D4RL locomotion offline | normalized score total: 1000.8 -> 1116.7 | Helped |
| add selective state-adaptive regularization | CQL · D4RL locomotion offline | normalized score total: 1030.4 -> 1139.1 | Helped |
| add selective state-adaptive regularization | TD3+BC · D4RL AntMaze offline | normalized score total: 131.8 -> 276.0 | Helped |
| add selective state-adaptive regularization | CQL · D4RL AntMaze offline | normalized score total: 294.1 -> 406.8 | Helped |
| use state-adaptive regularization with coefficient annealing | CQL · D4RL locomotion online fine-tuning, 250k steps | normalized score total: 1068.4 -> 1278.4 | Helped |
| use state-adaptive regularization with coefficient annealing | TD3+BC · D4RL AntMaze online fine-tuning, 250k steps | normalized score total: 112.8 -> 389.2 | Mixed |
| replace the global fixed coefficient with state-adaptive coefficients (ablation) | selective state-adaptive regularization · D4RL offline benchmarks | normalized score | Helped |
| select trajectories with high returns instead of using IQL-style state-level selection (ablation) | selective state-adaptive regularization · wide-ranging-distribution datasets | normalized score | Hurt |
| use IQL-style state-level selection rather than high-return trajectory selection (ablation) | selective state-adaptive regularization · wide-ranging-distribution datasets | normalized score | Helped |
| initialize the online replay buffer with no offline data instead of the selected subset (ablation) | CQL(SA) · D4RL locomotion online fine-tuning | normalized score total: 1279.4 -> 1190.1 | Hurt |
| add state-adaptive regularization | TD3+BC · training | training time (h): 2.25 -> 2.69 | Hurt |
Surprising findings
- Discarding the offline dataset entirely during online fine-tuning still gives competitive performance on non-sparse reward tasks, according to the appendix experiments.
Limitations
- Selective regularization needs return thresholds or IQL pretraining, which are task-specific hyperparameters; AntMaze subsets are chosen by trajectory success.
- Sensitivity to sub-dataset selection: high-return trajectory filtering degrades performance on heterogeneous replay datasets, motivating the IQL-style variant.
- Some settings are reported with high variance (e.g., AntMaze medium/large with std over 15) and only four seeds.
Experimental setup
Datasets: D4RL halfcheetah D4RL hopper D4RL walker2d D4RL antmaze
Benchmarks: D4RL MuJoCo D4RL AntMaze
Metrics: D4RL normalized score
Baselines: CQL TD3+BC IQL SPOT FamCQL AWAC Cal-QL
Builds on: CQL TD3+BC IQL SAC FamO2O
Closest papers by meaning
- An Implicit Trust Region Approach to Behavior Regularized Offline Reinforcement Learning (AAAI 2024 · similarity 0.92)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.91)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.90)
- Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning (ICML 2024 · similarity 0.90)
- State Entropy Regularization for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.