Research map / Reinforcement learning / Offline RL
Relaxed Stationary Distribution Correction Estimation for Improved Offline Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PORelDICE, which relaxes the positivity constraint in f-divergence-regularized implicit policy optimization (DICE) and adds an auxiliary function to reduce bias, improving offline RL on D4RL.
Problem
Implicit policy optimization via stationary distribution correction estimation (DICE) underperforms in offline RL, especially on large environments and low-quality datasets, due to biased estimates and the zero-gradient shape of the f-divergence conjugate function.
Approach
Relax the positivity constraint on the state-action visitation ratio to a lower bound epsilon (<0), giving a conjugate function with a non-decreasing lower tail and lower-bounded gradient, and add an auxiliary function U(s,a) to alleviate the optimistic-transition bias; the relaxation also lets low-advantage samples contribute to the value update.
Key results
- PORelDICE achieves the highest score on 7 of 9 Gym-MuJoCo tasks, with a Gym-MuJoCo total of 733.8 versus 600.4 for OptiDICE.
- On Antmaze-large, PORelDICE scores 50.4 on the playpen dataset and 43.6 on the diverse dataset, where OptiDICE scores 0.
- PORelDICE gives a Kitchen total of 188.0, the highest among the compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply PORelDICE instead of SQL (w/ trick) | Offline RL on Gym-Mujoco · Gym-Mujoco-total | normalized score: 703.2 -> 733.8 | Helped |
| Apply PORelDICE instead of IQL | Offline RL on Antmaze · Antmaze-total | normalized score: 382.5 -> 360.0 | Hurt |
| Apply PORelDICE instead of SQL (w/ trick) | Offline RL on Kitchen · Kitchen-total | normalized score: 181.0 -> 188.0 | Helped |
| Add the SQL trick to SQL (ablation) | SQL on Gym-Mujoco · Gym-Mujoco-total | normalized score: 668.8 -> 703.2 | Helped |
| Add the SQL trick to SQL (ablation) | SQL on Antmaze · Antmaze-total | normalized score: 351.4 -> 353.8 | Helped |
| Add the SQL trick to SQL (ablation) | SQL on Kitchen · Kitchen-total | normalized score: 178.0 -> 181.0 | Helped |
Surprising findings
- The surrogate objective used in OptiDICE is biased toward high-advantage transitions, and the auxiliary-function fix alone diverges because the non-decreasing conjugate causes nu to only increase.
- OptiDICE's policy in a toy example prefers a suboptimal state due to optimistic-transition bias from the surrogate objective.
Experimental setup
Datasets: D4RL Gym-MuJoCo D4RL Antmaze D4RL Kitchen
Benchmarks: D4RL Gym-MuJoCo D4RL Antmaze D4RL Kitchen
Metrics: normalized score return learning curves
Baselines: BC CQL IQL SQL (w/ trick) SQL (w/o trick) OptiDICE
Builds on: OptiDICE AlgaeDICE Sparse Q-learning Implicit Q-learning
Closest papers by meaning
- Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning (ICML 2022 · similarity 0.93)
- OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation (ICML 2021 · similarity 0.93)
- ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update (ICLR 2024 · similarity 0.92)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.