Research map / Reinforcement learning / Offline RL
An Implicit Trust Region Approach to Behavior Regularized Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes iTRPO, which adds a scaled log-probability ratio between the learned and behavior policy as a reward shaping term in BRAC, implicitly imposing a trust region between successive policies for offline RL.
Problem
Behavior regularization in offline RL (e.g., BRAC) keeps the learned policy close to the behavior policy but does not guarantee monotonic policy improvement, so the policy can wander within the feasible region and suffer from unstable learning under value estimation errors.
Approach
Reshape the immediate reward within the BRAC framework by a scaled log-probability ratio between the current learned policy and the behavior policy; this is shown to be equivalent to an extra KL trust region between successive policies plus the original behavior KL constraint, at negligible computational cost.
Key results
- iTRPO averages 79.3 normalized score across 15 MuJoCo locomotion D4RL tasks, versus 74.4 for the second-best PBRL and 69.4 for IQL.
- iTRPO achieves the highest score on 11 of 15 MuJoCo tasks, with reduced variance (±0.9) across tasks.
- On AntMaze-v2 six tasks, iTRPO totals 418.5 versus 392.4 for IQL and 359.6 for SPOT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add log-probability-ratio reward shaping to BRAC (ablation) | BRAC / iTRPO · MuJoCo Locomotion (15 tasks) | normalized score | Helped |
| add implicit trust-region optimization through reward shaping | iTRPO · MuJoCo Locomotion (15 tasks) | average normalized score: 74.4±5.3 -> 79.3±0.9 | Helped |
| add implicit trust-region optimization through reward shaping | iTRPO · Antmaze (6 tasks) | total normalized score: 392.4 -> 418.5 | Helped |
Surprising findings
- Behavior-regularized methods like BRAC can wander within the constraint region without improving, so a KL constraint to the behavior policy alone is insufficient for policy improvement.
- The mean shaping reward is usually negative, so iTRPO learns a more conservative value function than the original BRAC.
Limitations
- Theoretical guarantees rely on approximate value iteration and a concentrability assumption that may be hard to satisfy.
- Evaluated only on D4RL MuJoCo and AntMaze; scores for some baselines are taken from prior papers rather than re-run.
Experimental setup
Datasets: D4RL MuJoCo locomotion (v2) D4RL AntMaze (v2)
Benchmarks: D4RL MuJoCo locomotion D4RL AntMaze
Metrics: normalized score KL divergence estimation percentage difference (ablation)
Baselines: BEAR UWAC IQL TD3-BC CQL PBRL PLAS SPOT
Builds on: BRAC Munchausen value iteration TRPO D4RL
Closest papers by meaning
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.93)
- Learning to Trust Bellman Updates: Selective State-Adaptive Regularization for Offline RL (ICML 2025 · similarity 0.92)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.91)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.91)
- In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning (AAAI 2025 · similarity 0.89)
- Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.