Research map / Reinforcement learning / Imitation learning
Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TDIL, an IRL method for single-demonstration imitation learning that uses a learned transition discriminator to build a dense, dynamics-aware surrogate reward, reaching expert-level performance on MuJoCo and Adroit Door.
Problem
Single-demonstration imitation learning, where only one expert trajectory is available and no ground-truth reward exists; basic IRL yields sparse rewards and geometric dense rewards (L2, cosine, OT) ignore environment dynamics.
Approach
Defines expert proximity as states that can reach expert states in one step and learns a transition discriminator (trained on agent replay-buffer transitions with easy and reversed negatives) to compute a surrogate reward summing reachability to expert states, combined with GAIL reward and optionally BC loss, trained with SAC.
Key results
- TDIL (Ours R_TDIL + BC) reaches expert-level returns on all five MuJoCo tasks, e.g., Walker2d-v3 6,312 ± 47 vs expert 6,123 and Ant-v3 6,434 vs expert 6,561 (Ant with R_tot reaches 6,837).
- On AdroitHandDoor, both TDIL variants achieve 100% success while f-IRL and CFIL stay within 40% (mostly 0%).
- Relative return correlates positively with ground-truth return for blind checkpoint selection, while raw return does not consistently track it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TDIL (Rtot) instead of the strongest reported baseline | single-demo imitation learning · HalfCheetah-v3 | return: 14,560 $\pm$ 823 -> 15,624 $\pm$ 119 | Helped |
| use TDIL (Rtot) instead of the strongest reported baseline | single-demo imitation learning · Hopper-v3 | return: 4,131 $\pm$ 34 -> 4,115 $\pm$ 14 | No clear effect |
| use TDIL (Rtot) instead of the strongest reported baseline | single-demo imitation learning · Ant-v3 | return: 5,812 $\pm$ 2,692 -> 6,837 $\pm$ 19 | Helped |
| use TDIL (Rtot) instead of the strongest reported baseline | single-demo imitation learning · Humanoid-v3 | return: 5,499 $\pm$ 94 -> 6,302 $\pm$ 136 | Helped |
| use TDIL (Rtot) instead of the strongest reported baseline | single-demo imitation learning · Walker2d-v3 | return: 4,746 $\pm$ 316 -> 6334 $\pm$ 10 | Helped |
| add BC loss to the pure surrogate-reward variant (ablation) | TDIL · HalfCheetah-v3 | return: 12,630 $\pm$ 6,854 -> 15,666 $\pm$ 85 | Helped |
| add BC loss to the pure surrogate-reward variant (ablation) | TDIL · Hopper-v3 | return: 3,890 $\pm$ 562 -> 4,115 $\pm$ 14 | Helped |
| add BC loss to the pure surrogate-reward variant (ablation) | TDIL · Ant-v3 | return: 3,995 $\pm$ 2,408 -> 6,434 $\pm$ 66 | Helped |
| add BC loss to the pure surrogate-reward variant (ablation) | TDIL · Humanoid-v3 | return: 5,575$\pm$196 -> 5,758 $\pm$ 173 | Helped |
| add BC loss to the pure surrogate-reward variant (ablation) | TDIL · Walker2d-v3 | return: 6,281 $\pm$ 76 -> 6,312 $\pm$ 47 | No clear effect |
| remove reversed-transition hard negative samples (ablation) | TDIL · HalfCheetah-v3 | return: 15,666 $\pm$ 85 -> 15,718 $\pm$ 179 | No clear effect |
| remove reversed-transition hard negative samples (ablation) | TDIL · Hopper-v3 | return: 4,115 $\pm$ 14 -> 4,143 $\pm$ 6 | No clear effect |
Surprising findings
- Raw surrogate return initially rises then falls during training in Hopper, Ant, and Walker2d, reflecting improving discriminator accuracy rather than policy degradation.
- Pure surrogate reward without BC loss performs worse, but still matches or exceeds baselines in Walker2d-v3 where all baselines fail.
Limitations
- Single-demonstration results are reported on five MuJoCo tasks and one Adroit task; generalization to higher-dimensional or real-world settings is untested.
- Adroit experiments exclude other Adroit tasks because goal-conditioned tasks cannot be learned from one demonstration.
- Policy assumption requires BC or GAIL-based action prior; pure surrogate variant underperforms, indicating reliance on expert action learning.
Experimental setup
Datasets: MuJoCo expert demonstrations from a well-trained SAC agent (single trajectory) AdroitHandDoor-v1 expert demonstration
Benchmarks: HalfCheetah-v3 Hopper-v3 Ant-v3 Humanoid-v3 Walker2d-v3 AdroitHandDoor-v1
Metrics: ground-truth episodic return normalized performance vs expert success rate transition discriminator accuracy correlation for blind model selection
Baselines: BC GAIL f-IRL PWIL CFIL
Builds on: GAIL Soft Actor-Critic PWIL f-IRL CFIL
Closest papers by meaning
- Learning from Demonstrations via Capability-Aware Goal Sampling (NeurIPS 2025 · similarity 0.90)
- Limited Preference Aided Imitation Learning from Imperfect Demonstrations (ICML 2024 · similarity 0.89)
- Imitation Learning from Observation with Automatic Discount Scheduling (ICLR 2024 · similarity 0.88)
- GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation (AAAI 2024 · similarity 0.88)
- Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations (NeurIPS 2025 · similarity 0.88)
- Causal Imitation for Markov Decision Processes: a Partial Identification Approach (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.