Research map / Reinforcement learning / Imitation learning
Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PAGAR, a semi-supervised IRL-based imitation framework that trains a protagonist policy adversarially against an antagonist over a set of candidate reward functions consistent with expert demonstrations, to target task alignment rather than data alignment.
Problem
IRL-based imitation learning infers reward functions that match demonstrations but often fail to capture the true task objective, causing reward ambiguity, misspecification and hacking.
Approach
Treat expert demonstrations as weak supervision to build a delta-optimal reward set R_{E,delta}, then minimax-train a protagonist policy against an antagonist policy via protagonist-antagonist induced regret so the policy performs well across all candidate rewards rather than one IRL-optimal reward.
Key results
- PAGAR-VAIL reaches high performance with 1 demonstration on MiniGrid where VAIL with 1 demo performs much worse, with significantly higher sample efficiency.
- In zero-shot transfer to SimpleCrossingS9N2, SimpleCrossingS9N3 and FourRooms trained from SimpleCrossingS9N1 demos, PAGAR-based IL outperforms GAIL, VAIL and IQ-Learn.
- Combined with RECOIL on D4RL offline tasks, PAGAR-RECOIL improves hopper-random (111.16 vs 106.87), halfcheetah-random (92.94 vs 80.84) and ant-random (121 vs 113.34).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine PAGAR with RECOIL | RECOIL · hopper-random | return: 106.87\pm 2.69 -> 111.16\pm 0.51 | Helped |
| Combine PAGAR with RECOIL | RECOIL · halfcheetah-random | return: 80.84\pm 17.62 -> 92.94\pm 0.10 | Helped |
| Combine PAGAR with RECOIL | RECOIL · walker2d-random | return: 108.40\pm 0.04 -> 108.40 \pm 0.12 | No clear effect |
| Combine PAGAR with RECOIL | RECOIL · ant-random | return: 113.34\pm 2.78 -> 121\pm 5.86 | Helped |
Surprising findings
- Walker2d-random with PAGAR-RECOIL matches RECOIL (108.40) with no gain, indicating the method does not help uniformly across tasks.
Limitations
- Theoretical guarantees rely on strong assumptions (Lipschitz rewards, Wasserstein distance bounds, convexity-type conditions) that are hard to verify in practice.
- The delta/k hyperparameter that determines the candidate reward set must be chosen empirically and is not known a priori.
Experimental setup
Datasets: MiniGrid DoorKey-6x6-v0 MiniGrid SimpleCrossingS9N1-v0 D4RL hopper, halfcheetah, walker2d, ant expert and random datasets
Benchmarks: MiniGrid DoorKey-6x6 MiniGrid SimpleCrossingS9N1 SimpleCrossingS9N2 SimpleCrossingS9N3 FourRooms MuJoCo Ant MuJoCo Hopper MuJoCo Walker2d MuJoCo HalfCheetah
Metrics: episode return under default environment reward sample efficiency
Baselines: GAIL VAIL IQ-Learn f-IRL RECOIL
Builds on: GAIL AIRL unsupervised environment design PPO f-IRL
Closest papers by meaning
- Inverse Reinforcement Learning by Estimating Expertise of Demonstrators (AAAI 2025 · similarity 0.91)
- Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.91)
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.90)
- Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching (ICLR 2025 · similarity 0.90)
- Imitation Learning from Observation with Automatic Discount Scheduling (ICLR 2024 · similarity 0.89)
- EvIL: Evolution Strategies for Generalisable Imitation Learning (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.