Research map / Reinforcement learning / Offline RL
Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Conservative Policy Iteration (CPI) and CPI-RE for offline RL, iteratively refining the behavior-regularization reference policy on top of TD3+BC, with a tabular in-sample optimality guarantee and D4RL results.
Problem
Behavior regularization in offline RL performs poorly when the dataset comes from suboptimal policies, since its reference (behavior) policy is fixed and may be weak; it also fails to stitch suboptimal trajectories into in-sample optimal behavior.
Approach
Use conservative policy iteration: the reference policy for KL behavior regularization is iteratively replaced by the previous policy iterate (starting from the behavior policy), so policy improvement is guaranteed while staying on the dataset support; a CPI-RE variant keeps two actors and uses the currently better one as reference.
Key results
- CPI achieves total normalized score 1761.2 on Gym-MuJoCo+Antmaze+Adroit-style table vs. 1520.8 for IQL and 1747.6 for Diffusion-QL (Table 1, total across tasks); CPI-RE reaches 1803.9.
- On Antmaze, CPI-RE averages 438.8 total vs. 378.0 for IQL and 452.5 for Diffusion-QL, competitive with top methods.
- CPI runs at 8.5 s/epoch and 1.4 GB GPU memory versus 39.8 s/epoch for Diffusion-QL and 19.6 s/epoch for EDAC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| iteratively refine the behavior-regularization reference policy | TD3+BC-style offline RL · halfcheetah-medium | average normalized score: 58.8 -> 64.4 | Helped |
| add a reference-policy ensemble (ablation) | CPI · Antmaze | average normalized score: 416.0 -> 438.8 | Helped |
| add a reference-policy ensemble (ablation) | CPI · Adroit | average normalized score: 151.9 -> 157.7 | Helped |
| remove behavior regularization (ablation) | CPI · MuJoCo medium and medium-replay tasks | normalized score | Hurt |
| use forward KL instead of reverse KL (ablation) | CPI · hopper-medium, hopper-medium-replay, walker2d-medium, and walker2d-medium-replay | normalized score | Hurt |
| use two actor policies instead of one (ablation) | CPI-RE · halfcheetah-medium and halfcheetah-medium-replay | normalized score | Helped |
| increase the actor count from 2 to 4 (ablation) | CPI-RE · halfcheetah-medium and halfcheetah-medium-replay | normalized score | No clear effect |
| increase the behavior-policy weight λ to 0.9 (ablation) | CPI · halfcheetah-medium and hopper-medium | normalized score | Mixed |
| set the behavior-policy weight λ to 0.1 (ablation) | CPI · halfcheetah-medium and hopper-medium | normalized score | Mixed |
| use CPI instead of TD3+BC with matched α | offline RL · Antmaze | average normalized score: 188.0 -> 416.0 | Helped |
| use CPI instead of TD3+BC with dynamically changed α | offline RL · Above Total | average normalized score: 1113.0 -> 1128.1 | Helped |
Surprising findings
- Naive iterative actor-critic without behavior regularization performs poorly, showing the KL-to-behavior term is needed for stability with function approximation.
- TD3+BC with matched alpha to CPI performs worse than CPI, suggesting gains are not merely from tuning the regularization strength.
Limitations
- Performance with function approximation depends on tuning two hyperparameters (tau and lambda).
- Theoretical guarantee is only proven for the tabular setting; practical variant lacks convergence analysis.
- Performance gains are not uniform; CPI slightly trails EDAC on several Mujoco tasks.
Experimental setup
Datasets: D4RL Gym-MuJoCo v2 D4RL Antmaze v0 D4RL Adroit v1 7x7 GridWorld (10k transitions) FourRoom (10k transitions)
Benchmarks: D4RL Gym-MuJoCo D4RL Antmaze D4RL Adroit 7x7 GridWorld FourRoom
Metrics: normalized score runtime per epoch (s) GPU memory (GB)
Baselines: DT TD3+BC CQL IQL POR EDAC Diffusion-QL InAC
Builds on: TD3+BC Politex IQL InAC
Closest papers by meaning
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.94)
- Policy Regularization with Dataset Constraint for Offline Reinforcement Learning (ICML 2023 · similarity 0.94)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.94)
- An Implicit Trust Region Approach to Behavior Regularized Offline Reinforcement Learning (AAAI 2024 · similarity 0.93)
- Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.92)
- Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.