Research map / Reinforcement learning / Imitation learning
Iterative Regularized Policy Optimization with Imperfect Demonstrations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IRPO, which iterates offline imitation learning and KL-regularized online RL, using the online-learned policy to roll out better demonstrations for the next iteration, to fine-tune policies from imperfect demonstrations on MuJoCo, Reach, and a fixed-wing UAV attitude task.
Problem
Fine-tuning policies learned from imperfect, scarce demonstrations with KL-regularized online RL is limited by over-constrained exploration and primacy bias, which cap the achievable return.
Approach
Iterate offline IL and online KL-regularized RL, using the online-learned policy as the demonstrator to roll out new trajectories that replace low-quality demonstrations (demonstration boosting), so the reference policy and demonstration quality improve each iteration.
Key results
- IRPO reaches 7678.4 return on Halfcheetah vs 6954.97 for the best baseline (TGRL/PROTO-style EMA row) and 4967.05 for BC.
- On the attitude control task IRPO achieves a 0.54 success rate versus 0.38 for the best fixed-KL baseline (AlphaStar).
- Average relative improvement over three baselines is 10.40% (Halfcheetah), 14.63% (Hopper), 38.72% (Reach), and 42.11% (attitude control).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace single-stage offline-to-online fine-tuning with iterative imitation learning and demonstration boosting | Policy learning from imperfect demonstrations · Halfcheetah | expected return: 6954.97±606.67 -> 7678.4±60.81 | Helped |
| Replace single-stage offline-to-online fine-tuning with iterative imitation learning and demonstration boosting | Policy learning from imperfect demonstrations · Hopper | expected return: 2656.11±222.7 -> 3044.7±48.65 | Helped |
| Replace single-stage offline-to-online fine-tuning with iterative imitation learning and demonstration boosting | Policy learning from imperfect demonstrations · Reach | expected return: -7.36±2.07 -> -5.32±0.43 | Helped |
| Replace single-stage offline-to-online fine-tuning with iterative imitation learning and demonstration boosting | Policy learning from imperfect demonstrations · Attitude control | success rate: 0.32±0.01 -> 0.54±0.01 | Helped |
| Reduce KL regularization strength from 100 to 10−2 (ablation) | Online policy optimization · Halfcheetah | expected return: 5708.93±80.10 -> 7392.39±897.89 | Helped |
| Reduce KL regularization strength from 10−1 to 10−3 (ablation) | Online policy optimization · Hopper | expected return: 2641.55±207.13 -> 1626.12±770.63 | Hurt |
| Reduce KL regularization strength from 10−1 to 10−2 (ablation) | Online policy optimization · Reach | expected return: -10.39±1.92 -> -2.89±0.13 | Helped |
| Reduce KL regularization strength from 10−1 to 10−2 (ablation) | Online policy optimization · Attitude control | success rate: 0.225±0.031 -> 0.362±0.018 | Helped |
| Roll out demonstrations with the online-learned policy for the next iteration instead of retaining only the policy (ablation) | IRPO · Attitude control | success rate | Helped |
| Increase the demonstration fraction from 10% to 50% and to all available demonstrations (ablation) | IRPO · Attitude control | success rate | Helped |
| Use demonstrations concentrated in the medium-difficulty goal region rather than easy or hard goals (ablation) | IRPO · Attitude control | success rate | Helped |
| Increase KL regularization strength as demonstration quality improves (ablation) | IRPO · Attitude control | success rate | Helped |
Surprising findings
- Remaining with the online-learned policy for the next iteration performs worse than rolling out new demonstrations, suggesting data is more valuable than the policy itself for the next iteration.
- Increasing KL regularization strength in the first iteration hurts; the best KL strength shifts to stronger values only as demonstration quality improves.
Limitations
- The indicator function for demonstration boosting relies on task understanding and is not well explored.
- Experiments are limited to on-policy RL; applicability to off-policy RL is unclear.
- Theoretical bounds rely on reward depending only on states and finite-horizon MDP assumptions that only fully or substantially hold for the tasks studied.
Experimental setup
Datasets: D4RL Halfcheetah D4RL Hopper Modified Reach (Bullet) fixed-wing UAV attitude control demonstrations (PID and human play)
Benchmarks: Halfcheetah (MuJoCo) Hopper (MuJoCo) Reach (PyBullet) fixed-wing UAV attitude control
Metrics: expected return success rate KL divergence
Baselines: BC AlphaStar-style fixed KL PIRLNav-style fixed KL Reincarnating RL annealed KL TGRL PROTO EMA regularization
Builds on: behavior cloning KL-regularized RL fine-tuning PROTO primacy bias (Nikishin et al. 2022) Reincarnating RL
Closest papers by meaning
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.88)
- Demonstration-Regularized RL (ICLR 2024 · similarity 0.88)
- Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations (NeurIPS 2025 · similarity 0.87)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.86)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.86)
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.