Research map / Reinforcement learning / Offline RL
Hybrid Reinforcement Learning from Offline Observation Alone
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FOOBAR, a forward-backward hybrid RL algorithm that learns from observation-only offline state data using only trace-model access, matching reset-model performance under an admissibility assumption.
Problem
Hybrid RL where offline data contains only states (observation-only), and the online environment offers only a trace model (reset to initial state) rather than a reset-to-any-state simulator.
Approach
Use forward state-moment-matching (FAIL-style adversarial imitation with a discriminator class built from the value function class) to learn a roll-in policy that matches the offline state distribution, then run PSDP backward with this roll-in as a surrogate reset model; under admissibility, this matches reset-model sample complexity.
Key results
- On combination lock, FOOBAR is much more sample efficient than online RL and its sample efficiency is comparable to HyQ, which uses canonical offline data with actions and rewards.
- On hammer-binary, FOOBAR's backward run is comparable to RLPD, though the forward run needs about 10x more samples than the backward run.
- Under benign inadmissible data, FOOBAR achieves relative success 1 like PSDP, but under the adversarial inadmissible construction it fails (0) while PSDP reaches 1.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Foobar with benign inadmissible offline data instead of the forward pass alone | Foobar · Combination lock, benign inadmissible data | relative success rate: $0.12~(0.1,0.135)$ -> $1~(1,1)$ | Helped |
| use Foobar rather than the forward pass alone | Foobar · Combination lock, adversarial inadmissible data | relative success rate: $0~(0,0)$ -> $0~(0,0)$ | No clear effect |
| use Foobar rather than PSDP with reset access | Combination lock · Combination lock, benign inadmissible data | relative success rate: $1~(1,1)$ -> $1~(1,1)$ | No clear effect |
| use Foobar rather than PSDP with reset access | Combination lock · Combination lock, adversarial inadmissible data | relative success rate: $1.1~(0.95,1.15)$ -> $0~(0,0)$ | Hurt |
| use Foobar rather than pure online RL | Hybrid RL on combination lock · Combination lock | sample efficiency | Helped |
| use Foobar rather than canonical hybrid Q-Learning | Hybrid RL on combination lock · Combination lock | sample efficiency | Mixed |
| use Foobar rather than RLPD | Hybrid RL on hammer-binary · Hammer-binary | performance | Hurt |
Surprising findings
- Under adversarial inadmissible offline data, FOOBAR fails completely while the reset-model PSDP succeeds, consistent with the hardness results.
Limitations
- Requires admissibility assumption; without it, trace-model hardness examples show exponential separation.
- Forward phase (state moment matching) is less sample efficient in practice, especially with high-dimensional continuous actions.
- Worse dependency on action space size A than reset-model methods; experiments use only two benchmarks.
Experimental setup
Datasets: D4RL hammer (binary reward) combination lock (epsilon-greedy pi* offline data)
Benchmarks: combination lock (H=100) hammer-binary (D4RL Adroit, H=50) inadmissible combination lock (H=10)
Metrics: evaluation return / success rate vs. sample count relative success rate over optimal policy
Baselines: HyQ BRIEE Online RL PSDP RLPD
Builds on: PSDP FAIL HyQ RLPD SAC
Closest papers by meaning
- Leveraging Offline Data in Online Reinforcement Learning (ICML 2023 · similarity 0.93)
- Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.91)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.91)
- Simple Ingredients for Offline Reinforcement Learning (ICML 2024 · similarity 0.91)
- Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees (ICLR 2024 · similarity 0.91)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.