Research map / Reinforcement learning / Offline RL
Rethinking Optimal Transport in Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reframes offline RL as a maximin optimal transport problem with Q as cost and uses partial OT (extremal OT) to map states to the best subset of dataset actions, yielding the Partial Policy Learning (PPL) algorithm.
Problem
Offline RL from datasets collected by mixtures of sub-optimal experts requires stitching the best behaviors; behavior cloning limits policy improvement and Wasserstein-based OT methods clone the whole dataset.
Approach
Views the offline RL policy as a transport map between state and action distributions with the critic as transport cost, and uses partial (inequality-constrained) OT so the policy maps each state only to the most relevant portion of the behavior action distribution; optimized via a maximin formulation with a potential network f >= 0 and no 1-Lipschitz constraint.
Key results
- On Antmaze-v2, PPL^R (ReBRAC-based) reaches a total of 520 vs 461.2 for ReBRAC and 384 for IQL-based OTR+IQL, with large-play improving from 60.4 to 76.8.
- On MuJoCo, PPL^R totals 799.45 across 9 datasets vs 796.9 for ReBRAC.
- On Android, PPL^R totals 711.44 vs 703.2 for ReBRAC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply partial policy learning on top of ReBRAC | ReBRAC · Antmaze-v2 | averaged normalized score (total): 461.2 -> 520 | Helped |
| Apply partial policy learning on top of ReBRAC | ReBRAC · MuJoCo-v2 | averaged normalized score (total): 796.9 -> 799.45 | Helped |
| Apply partial policy learning on top of ReBRAC | ReBRAC · Android-v1 | averaged normalized score (total): 703.2 -> 711.44 | Helped |
| Use PPL^R rather than ReBRAC | ReBRAC · Antmaze-v2 large-play | averaged normalized score: 60.4 -> 76.8 | Helped |
| Use PPL^R rather than ReBRAC | ReBRAC · Antmaze-v2 large-diverse | averaged normalized score: 54.4 -> 76.6 | Helped |
| Use PPL^R rather than ReBRAC | ReBRAC · MuJoCo-v2 | averaged normalized score | Mixed |
Surprising findings
- Directly minimizing -Q^beta(s,a) in the toy setting caused overestimation bias, with the policy predicting the maximal value 1.
Limitations
- The parameter w is task-dependent and must be tuned per task.
- Gains over ReBRAC on MuJoCo and Android are marginal; many results within error bars.
- Statistical significance not reported (checklist answer No).
Experimental setup
Datasets: D4RL MuJoCo D4RL Antmaze-v2 D4RL Android-v1
Benchmarks: D4RL MuJoCo D4RL Antmaze-v2 D4RL Android-v1 toy shortest-path task
Metrics: normalized score
Baselines: BC One-Step RL CQL IQL OTR+IQL TD3+BC ReBRAC
Builds on: TD3+BC ReBRAC One-Step RL Wasserstein BRAC extremal optimal transport
Closest papers by meaning
- Efficient Online Reinforcement Learning with Offline Data (ICML 2023 · similarity 0.91)
- Simple Ingredients for Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
- Online Optimization for Offline Safe Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Bayesian Design Principles for Offline-to-Online Reinforcement Learning (ICML 2024 · similarity 0.88)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.