Research map / Reinforcement learning / Policy gradient and actor-critic
OMPO: A Unified Framework for RL under Policy and Dynamics Shifts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OMPO, a unified online RL framework that matches transition occupancy distributions via a discriminator-based correction and a DICE-style min-max actor-critic, handling policy shifts, domain adaptation, and non-stationary dynamics.
Problem
Online RL learning from data collected under varying policies or dynamics (policy shifts, domain adaptation, non-stationary dynamics) yields distribution mismatch, biased policy evaluation, high variance, and existing methods are specialized to one shift type.
Approach
Formulates a surrogate RL objective based on transition occupancy (s,a,s') matching, estimates the discrepancy log-ratio with a GAN-style discriminator between a small local buffer (current on-policy data) and a global replay buffer, and converts the Bellman-flow constrained problem into a tractable min-max via Lagrangian duality and Fenchel conjugate, solved with a two-timescale actor-critic.
Key results
- Across 12 tasks in 3 shift settings, OMPO outperforms specialized baselines (SAC, TD3, AlgaeDICE, DARC, DR, CEMRL, CaDM) with lower variance, per the paper's summary.
- On 10 stochastic robot manipulation tasks, OMPO reaches e.g. 68.5% success on Coffer-Push vs 15.8% (SAC) and 3.3% (TD3), and 84.2% on Hammer vs 15.4% (SAC).
- OMPO-DR (OMPO combined with domain randomization) achieves strong target-domain performance, e.g. Walker2d nearly converges with about 60 target trajectories (60k steps).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use OMPO instead of SAC | Panda robot manipulation · Panda-Reach-Den | success rate: 92.6% -> 97.5% | Helped |
| use OMPO instead of SAC | Panda robot manipulation · Panda-Reach-Spr | success rate: 94.5% -> 93.1% | No clear effect |
| use OMPO instead of SAC | Meta-World manipulation · Coffer-Push | success rate: 15.8% -> 68.5% | Helped |
| use OMPO instead of SAC | Meta-World manipulation · Drawer-Open | success rate: 57.7% -> 93.4% | Helped |
| use OMPO instead of SAC | Meta-World manipulation · Door-Unlock | success rate: 93.5% -> 98.9% | Helped |
| use OMPO instead of SAC | Meta-World manipulation · Door-Open | success rate: 97.5% -> 99.5% | Helped |
| use OMPO instead of SAC | Meta-World manipulation · Hammer | success rate: 15.4% -> 84.2% | Helped |
| remove the distribution discriminator (ablation) | OMPO on Hopper · stationary and domain adaptation environments | learning performance | Hurt |
| use a smaller local buffer (ablation) | OMPO on Hopper non-stationary environments · Hopper non-stationary | performance stability | Mixed |
| use a larger local buffer (ablation) | OMPO on Hopper non-stationary environments · Hopper non-stationary | policy optimization performance | Hurt |
| increase the weighted factor α excessively (ablation) | OMPO on Hopper non-stationary environments · Hopper non-stationary | policy performance | Hurt |
| decrease the weighted factor α excessively (ablation) | OMPO on Hopper non-stationary environments · Hopper non-stationary | shift handling | Hurt |
Surprising findings
- A variant of OMPO without the distribution discriminator performs comparably to SAC in stationary settings and suffers high variance under domain adaptation.
Limitations
- Requires strictly positive rewards (reward mapped to (0, r_max]) for the log-reward objective, which the authors flag as an assumption to relax.
- Performance is sensitive to local buffer size: small buffers cause discriminator instability, large buffers lose on-policy freshness.
- Relies on a local consistency assumption for non-stationary dynamics.
Experimental setup
Datasets: OpenAI Gym (Hopper-v3, Walker2d-v3, Ant-v3, Humanoid-v3) Meta-World Panda Robots
Benchmarks: OpenAI Gym Hopper-v3 Walker2d-v3 Ant-v3 Humanoid-v3 Meta-World Panda-Reach Panda-Push
Metrics: episodic return success rate
Baselines: SAC TD3 AlgaeDICE DARC Domain Randomization CEMRL CaDM
Builds on: DICE SAC DARC AlgaeDICE
Closest papers by meaning
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.89)
- ODRL: A Benchmark for Off-Dynamics Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization (ICLR 2024 · similarity 0.89)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.89)
- Occupancy-based Policy Gradient: Estimation, Convergence, and Optimality (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.