Research map / Reinforcement learning / Offline RL
ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes O-DICE, which projects the backward gradient of V-DICE orthogonal to the forward gradient, giving state-action-level behavior regularization and SOTA offline RL and IL results on D4RL.
Problem
DICE-based offline RL and imitation learning methods enforce state-action-level behavior constraints in principle but perform much worse than action-constrained SOTA methods, because the backward (next-state) gradient in the true-gradient value update interferes with the forward gradient.
Approach
Project the backward gradient onto the normal plane of the forward gradient (orthogonal-gradient update) inside V-DICE; this keeps the action-level effect of the forward gradient while adding state-level regularization that alleviates feature co-adaptation.
Key results
- O-DICE achieves the highest score on 13 of 15 D4RL tasks in the main table, e.g., hopper-m-r 99.9 vs 98.1 for EQL and antmaze-m-p 86.0 vs 77.5 for EQL.
- O-DICE attains much smaller percent differences in worst-episode performance than EQL and S-DICE, indicating better policy robustness.
- On offline IL with one expert trajectory, O-DICE beats BC, ValueDICE and IQLearn on most of the eight Hopper and Walker2d trajectories (e.g., Walker2d Traj 1: 75.6 vs 64.6 for BC).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use orthogonal-gradient updates instead of true-gradient updates (ablation) | V-DICE (O-DICE vs OptiDICE) · D4RL benchmarks | normalized score | Helped |
| Use orthogonal-gradient updates instead of semi-gradient updates (ablation) | V-DICE (O-DICE vs S-DICE) · D4RL benchmarks | normalized score | Helped |
| Apply orthogonal-gradient updates | V-DICE · hopper-m | normalized score: 61.5 ±2.3 -> 86.1 ±4.0 | Helped |
| Apply orthogonal-gradient updates | V-DICE · antmaze-m-p | normalized score: 50.6 ±11.1 -> 86.0 ±1.6 | Helped |
| Apply orthogonal-gradient updates | V-DICE · halfcheetah-m | normalized score: 47.3 ±0.2 -> 47.4 ±0.2 | No clear effect |
| Apply orthogonal-gradient updates | V-DICE · walker2d-m-r | normalized score: 37.3 ±5.1 -> 83.6 ±4.1 | Helped |
| Apply orthogonal-gradient updates | V-DICE · Hopper-expert trajectories | normalized return | Mixed |
| Apply orthogonal-gradient updates (ablation) | Q-DICE toy example · grid-world navigation | policy quality | Helped |
| Use semi-gradient updates instead of orthogonal-gradient updates (ablation) | V-DICE toy example · grid-world navigation | OOD-state robustness | Hurt |
| Use true-gradient updates instead of orthogonal-gradient updates (ablation) | V-DICE toy example · grid-world navigation | policy quality | Hurt |
| Apply orthogonal-gradient updates | O-DICE · D4RL evaluation episodes | percent difference of worst episode | Helped |
| Apply orthogonal-gradient updates | O-DICE · offline IL tasks | normalized return | Mixed |
Surprising findings
- O-DICE does not need the double Q-learning trick, unlike almost all prior offline RL algorithms.
- Plain true-gradient DICE (OptiDICE) performs much worse than semi-gradient variants, with the backward gradient cancelling the forward gradient's effect.
Limitations
- Backward gradient introduces a double-sampling issue in stochastic environments.
- O-DICE has two hyperparameters (alpha/lambda and eta) to tune.
Experimental setup
Datasets: D4RL halfcheetah, hopper, walker2d (medium, medium-replay, medium-expert) D4RL antmaze (umaze, medium, large variants) D4RL expert datasets (for IL trajectories)
Benchmarks: D4RL MuJoCo D4RL AntMaze offline imitation learning on D4RL expert trajectories grid world toy example
Metrics: normalized score percent difference of worst episode feature dot product
Baselines: TD3+BC CQL IQL EQL OptiDICE f-DVL S-DICE BC ValueDICE IQLearn
Builds on: V-DICE OptiDICE Exponential Q-Learning true-gradient update semi-gradient update
Closest papers by meaning
- Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.92)
- Relaxed Stationary Distribution Correction Estimation for Improved Offline Policy Optimization (AAAI 2024 · similarity 0.92)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.89)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.