Research map / Reinforcement learning / Imitation learning
Blending Imitation and Reinforcement Learning for Robust Policy Improvement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RPI, a policy-gradient method that adaptively blends imitation from multiple black-box suboptimal oracles and RL on the learner's own value function, using confidence-aware active policy selection (RAPS) and a robust policy gradient (RPG).
Problem
Sample-efficient RL with suboptimal, black-box oracles: existing IL+RL methods (MAMBA, MAPS, LOKI) keep imitating poor oracles even when the learner is better, making policy improvement non-robust.
Approach
Extends the oracle set with the learner's current policy, builds a max-plus baseline over oracle and learner value functions, and uses UCB/LCB-based active policy selection plus a confidence-thresholded GAE-style advantage so the agent imitates oracles only in states where they outperform the learner and otherwise self-improves.
Key results
- RPI outperforms PPO-GAE, Max-Aggregation, LOKI-variant, MAMBA, and MAPS across all eight tasks with three diversified oracles, per the main results figure.
- On Cartpole, RPI reaches a return of 645 with three oracles versus below 600 with a single oracle.
- RPI-LCB/UCB outperforms RPI-MEAN across all benchmarks by an overall margin of 40%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| blend imitation learning and reinforcement learning using RPI | RPI · Cheetah-run, CartPole-swingup, Pendulum-swingup, Walker-walk, Window-close, Button-press, Faucet-open, and Drawer-close | best return | Helped |
| give RPI access to three diversified oracles instead of only one (ablation) | RPI · Cartpole | return: below 600 -> 645 | Helped |
| use RAPS, which can select the learner to roll out, instead of APS without the learner in RIRO (ablation) | RPI · Cartpole | return | Helped |
| use a confidence threshold on oracle value estimates in RPG (ablation) | RPI · Cartpole | return | Helped |
| use LCB/UCB uncertainty-aware policy selection instead of mean estimates (ablation) | RPI · All benchmarks | overall performance: 40% | Helped |
Surprising findings
- PPO-GAE outperforms IL-based methods in later training stages on most dense-reward environments, while IL methods are better in sparse-reward domains.
Limitations
- Evaluated only on eight continuous-control tasks with black-box oracles from PPO/SAC; no discrete or language settings.
- Uses a fixed confidence threshold (0.5) and ensemble size of five, with threshold sensitivity studied only in appendix.
- Authors note future need to handle missing state or oracle information.
Experimental setup
Datasets: DeepMind Control Suite Meta-World
Benchmarks: DeepMind Control Cheetah-run CartPole-swingup Pendulum-swingup Walker-walk Meta-World Window-close Button-press Faucet-open Drawer-close
Metrics: best return standard error over trials
Baselines: PPO-GAE Max-Aggregation LOKI-variant MAMBA MAPS best oracle
Builds on: MAMBA MAPS LOKI AggreVaTeD PPO generalized advantage estimation
Closest papers by meaning
- Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.90)
- Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data (AAAI 2025 · similarity 0.90)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.89)
- Adversarial Diffusion for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Simplifying Constraint Inference with Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.