Research map / Reinforcement learning / Offline RL
Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Uni-O4, an on-policy PPO-based framework that unifies offline and online RL by using ensemble behavior cloning with disagreement regularization, multi-step offline policy improvement gated by a model-based OPE (AM-Q), and standard online PPO fine-tuning.
Problem
Offline-to-online RL typically treats offline and online learning as separate procedures; offline-trained conservatism causes fine-tuning instability and asymptotic suboptimality, and existing offline policy improvement (e.g., BPPO) relies on online evaluation.
Approach
Use a single on-policy PPO objective for both offline and online phases; learn a diverse ensemble of behavior policies via BC with a disagreement penalty; perform multi-step offline policy improvement with a simple model-based offline policy evaluation (AM-Q) instead of online evaluation; then fine-tune with standard online PPO without extra conservatism.
Key results
- On D4RL Gym locomotion, Uni-O4 reaches 816.4 total vs 818.9 for ATAC and 751.0 for BPPO, and achieves 1322.0 total across locomotion, Adroit, and Kitchen versus 1253.4 for BPPO.
- On Antmaze (6 tasks), Uni-O4 totals 447.9 versus 378.0 for IQL and 321.0 for GLS-style RvS-G baseline.
- AM-Q offline policy evaluation reaches about 80% accuracy, and about 95% when allowing a 20% error margin, on MuJoCo tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Uni-O4 instead of ATAC | offline RL on D4RL MuJoCo locomotion tasks · D4RL MuJoCo locomotion total | normalized return: 818.9 vs 816.4 ± 10.5 | No clear effect |
| Use Uni-O4 instead of BPPO | offline RL on D4RL Adroit tasks · D4RL Adroit total | normalized return: 291.4 vs 288.6 ± 40.0 | No clear effect |
| Use Uni-O4 instead of BPPO | offline RL on D4RL Kitchen tasks · D4RL Kitchen total | normalized return: 211.0 -> 216.9 ± 10.7 | Helped |
| Use Uni-O4 instead of BPPO | offline RL across D4RL tasks · D4RL total | normalized return: 1253.4 -> 1322.0 ± 61.2 | Helped |
| Use Uni-O4 instead of IQL | offline RL on Antmaze tasks · Antmaze total | normalized return: 378.0 -> 447.9 ± 28.1 | Helped |
| Add a minor disagreement penalty to behavior cloning (ablation) | Uni-O4 behavior-policy initialization · MuJoCo tasks | normalized return | Helped |
| Increase the disagreement penalty beyond a minor value (ablation) | Uni-O4 behavior-policy initialization · MuJoCo tasks | normalized return | Hurt |
| Add a disagreement penalty rather than use no penalty (ablation) | Uni-O4 behavior-policy initialization · MuJoCo tasks | normalized return | Helped |
| Vary ensemble size; select 4 policies (ablation) | Uni-O4 behavior-policy initialization · MuJoCo tasks | normalized return | Mixed |
| Use AM-Q offline policy evaluation in place of online evaluation (ablation) | Uni-O4 multi-step policy improvement · MuJoCo tasks | OPE accuracy: approximately 80%; 95% | Helped |
| Fine-tune from Uni-O4 offline initialization rather than PPO from scratch | MuJoCo locomotion environments · MuJoCo locomotion | normalized return | Helped |
Surprising findings
- Online PPO from scratch plateaus after surpassing offline performance, whereas Uni-O4 fine-tuning keeps improving rapidly.
- Q-ensemble method Off2on requires about 18 hours versus about 30 minutes for Uni-O4 on the same fine-tuning setting.
Limitations
- Offline policy evaluation accuracy is only about 80% on MuJoCo tasks, relying on a learned dynamics model.
- Real-world evaluation is limited to a single quadruped robot and latex mattress terrain.
- Ensemble of policies adds training overhead relative to a single policy.
Experimental setup
Datasets: D4RL MuJoCo locomotion D4RL Adroit D4RL Kitchen D4RL Antmaze real-world Go1 latex mattress rollouts
Benchmarks: D4RL MuJoCo locomotion D4RL Adroit D4RL Kitchen D4RL Antmaze real-world Go1 quadruped tasks
Metrics: normalized return OPE accuracy average return over 5 trials
Baselines: CQL TD3+BC Onestep RL IQL COMBO BPPO ATAC Off2on PEX Cal-ql walk these ways
Builds on: PPO BPPO IQL Behavior cloning D4RL
Closest papers by meaning
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.92)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.91)
- Efficient Online Reinforcement Learning with Offline Data (ICML 2023 · similarity 0.91)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
- ODRL: A Benchmark for Off-Dynamics Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RL (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.