Research map / Reinforcement learning / Offline RL
Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hybrid Actor-Critic (HAC) and Hybrid NPG (HNPG), which combine on-policy NPG/policy gradient with off-policy fitted policy evaluation on offline data, giving doubly robust guarantees under hybrid RL.
Problem
Hybrid RL (offline plus online data) typically relies on off-policy/Bellman-backup learning, which needs strong assumptions such as Bellman completeness and can be unstable; the question is whether on-policy policy gradient can be combined with offline data to get the best of both worlds.
Approach
Augment on-policy NPG-style policy evaluation with a TD loss on offline data plus an on-policy Monte-Carlo regression loss on online data, so the critic is accurate on the on-policy distribution regardless of whether Bellman completeness holds, while still exploiting offline data when it does.
Key results
- HNPG solves continuous Comblock up to horizon 50 with about 30m online samples, whereas TRPO fails even at horizon 5 and RLPD only solves up to horizon 15.
- On image-based Comblock, HNPG reliably solves up to horizon 30 with about 10m online samples, while RLPD fails at horizon 10.
- HNPG achieves about 0.8 test reward versus about 0.6 for RLPD at horizon 5 on image-based Comblock.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine on-policy learning with offline-data off-policy learning | HNPG versus TRPO · Continuous Comblock | maximum solved horizon: 5 -> 50 | Helped |
| Use on-policy learning together with offline-data learning instead of off-policy hybrid learning alone | HNPG versus RLPD · Continuous Comblock | maximum solved horizon: 15 -> 50 | Helped |
| Combine on-policy learning with offline-data off-policy learning | HNPG versus TRPO · Image-Based Continuous Comblock | maximum solved horizon: HNPG ... reliably solves up to horizon 30; TRPO fails to solve horizon 5 | Helped |
| Use on-policy learning together with offline-data learning instead of off-policy hybrid learning alone | HNPG versus RLPD · Image-Based Continuous Comblock | maximum solved horizon: RLPD ... completely fails for horizon 10; HNPG ... reliably solves up to horizon 30 | Helped |
| Use HNPG rather than RLPD | Image-Based Continuous Comblock · test environment | reward: around 0.6 -> around 0.8 | Helped |
| Use HNPG rather than RLPD | Image-Based Continuous Comblock · train and test learning curves | variance | Helped |
Surprising findings
- Off-policy hybrid RLPD's online TD critic loss explodes on image-based Comblock, while HNPG's supervised on-policy loss stays small.
Limitations
- Theoretical bounds depend on Bellman error transfer or concentrability coefficients that may be large; sharper rates not established.
- Experiments are limited to synthetic combination lock environments with a 10-dimensional continuous action space.
- Softmax-based HAC is intractable for large action spaces; practical method uses parameterized HNPG with approximations (no line-search-free theory).
Experimental setup
Datasets: CIFAR-100 rich-observation combination lock (synthetic)
Benchmarks: continuous Comblock image-based continuous Comblock (CIFAR-100 with CLIP features)
Metrics: sample complexity (online samples to reach 0.5 probability o train/test reward critic loss
Baselines: TRPO RLPD PPO BC CQL HyQ
Builds on: natural policy gradient fitted policy evaluation TRPO RLPD HyQ
Closest papers by meaning
- Offline RL Policies Should Be Trained to be Adaptive (ICML 2022 · similarity 0.91)
- Leveraging Offline Data in Online Reinforcement Learning (ICML 2023 · similarity 0.91)
- Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.91)
- Hybrid Reinforcement Learning from Offline Observation Alone (ICML 2024 · similarity 0.91)
- Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · similarity 0.90)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.