Research map / Reinforcement learning / Imitation learning

Limited Preference Aided Imitation Learning from Imperfect Demonstrations

ICML 2024 · Reinforcement learning › Imitation learning · approach: Offline imitation from imperfect data

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PAIL, an imitation learning method that learns a preference reward from limited human preference queries to reweight imperfect demonstrations and augments the demonstration set with high-preference-return trajectories explored by the policy.

Problem

Imitation learning from only imperfect (suboptimal, unlabeled mixed-quality) demonstrations cannot exceed demonstration quality; optimal demonstrations are often unavailable, and richer supervision such as scores or rankings is costly to obtain.

Approach

Learn a Bradley-Terry preference reward from a small number of preference queries, use it to reweight the imperfect demonstrations via a closed-form softmax distribution over preference returns, run max-causal-entropy IRL (SAC-based, DAC-style discriminator) on the reweighted data, and periodically add high-preference-return policy trajectories to the demonstration set so data quality keeps improv

Key results

• PAIL achieves 0.86 average normalized return on MuJoCo with 60 preference queries versus 0.49 for the best prior baseline, a 73.2% average improvement over prior SOTAs across all tasks (75.5% MuJoCo, 61.8% DMC).
• On GridWorld, PAIL reaches a 87.24% arrival rate versus 45.82% for MCE-IRL and under 1% for PEBBLE.
• PAIL outperforms the best initial demonstration trajectory in 16 of 18 task-dataset settings.

Measured effects reported in the paper

Surprising findings

• PEBBLE and BC-PEBBLE, which use learned preference rewards as RL reward, almost fail on GridWorld, indicating that preference rewards learned from few queries generalize poorly when used directly as RL reward.
• Even with only 2 preference queries, PAIL improves normalized return by almost 0.4 over the no-query case.

Limitations

• Assumes near-perfect preferences from a scripted teacher using oracle reward; noisy human preferences not addressed.
• For trajectories mixing good and bad segments, PAIL cannot exclusively use the good parts, limiting learning efficiency.
• Preference reward correlation with oracle reward drops substantially on replay-buffer samples, indicating limited extrapolation.

Experimental setup

Datasets: MuJoCo imperfect demonstrations (L, M, H quality) DMControl imperfect demonstrations (M quality) GridWorld imperfect demonstrations (M)

Benchmarks: GridWorld MuJoCo (Ant, HalfCheetah, Hopper, Humanoid, Walker2d) DMControl (cheetah run, quadruped walk, walker walk)

Metrics: arrival rate out rate normalized average return Pearson correlation between preference and oracle reward

Baselines: MCE-IRL SAIL-TRPO SAIL-SAC AILP PEBBLE BC-PEBBLE

Builds on: PEBBLE DAC SAC Bradley-Terry preference model SAIL

Closest papers by meaning

• Adversarial Imitation Learning with Preferences (ICLR 2023 · similarity 0.91)
• Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations (NeurIPS 2025 · similarity 0.89)
• Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration (ICLR 2025 · similarity 0.89)
• Predictive Preference Learning from Human Interventions (NeurIPS 2025 · similarity 0.89)
• Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learning (ICML 2024 · similarity 0.89)
• Efficient Imitation under Misspecification (ICLR 2025 · similarity 0.89)