Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Imitation Learning from Observation with Automatic Discount Scheduling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Automatic Discount Scheduling (ADS), which adaptively raises the RL discount factor as an agent progressively matches expert observation sequences, to fix proxy-reward ILfO failures on progress-dependent Meta-World tasks.
Problem
Imitation learning from observation (ILfO) with proxy rewards fails on tasks with progress dependency, where later proxy rewards hinder learning of earlier behaviors.
Approach
Dynamically schedule the RL discount factor based on a progress recognizer (LIS-based alignment with expert demos), starting with a small gamma so early rewards dominate and increasing gamma only as early behaviors are mastered; applied on top of optimal-transport proxy rewards.
Key results
- ADS on top of OT outperforms baseline OT in 7 of 9 Meta-World tasks in final success rate, and improves sample efficiency in 8 of 9 tasks, averaged over 8 seeds.
- ADS consistently outperforms constant discount factors (0.9, 0.93, 0.96, 0.99) and manually exponential discount schedules.
- Applying ADS to GAIfO significantly improves it on five of nine tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Automatic Discount Scheduling (ADS) to OT | OT · Meta-World, 9 tasks | task success rate | Helped |
| Add Automatic Discount Scheduling (ADS) to OT | OT · Meta-World, 9 tasks | sample efficiency | Helped |
| Add ADS to GAIfO | GAIfO · Meta-World, 9 tasks | performance | Helped |
| Use ADS instead of a constant discount factor (ablation) | OT · Meta-World tasks | performance | Helped |
| Use ADS instead of an exponential discount schedule (ablation) | OT · Meta-World tasks | performance | Helped |
| Vary progress-recognizer threshold λ among 0.8, 0.9, and 1.0 (ablation) | OT+ADS · Meta-World tasks | performance | No clear effect |
| Use smaller α in the discount mapping (ablation) | OT+ADS · Basketball and other early critical tasks | success rate | Helped |
| Use larger α in the discount mapping (ablation) | OT+ADS · Assembly and other later challenging tasks | success rate | Helped |
| Schedule discount factor using continuous expert-trajectory truncation instead of ADS (ablation) | OT · Meta-World tasks | performance | Mixed |
| Schedule discount factor using tiered expert-trajectory truncation instead of ADS (ablation) | OT · Meta-World tasks | performance | Mixed |
| Use LIS as the proxy reward (ablation) | RL with LIS rewards · Meta-World tasks | performance | Hurt |
| Compute costs with an R3M-pretrained ResNet-50 instead of the ImageNet-pretrained encoder (ablation) | OT+ADS · Meta-World tasks | performance | Hurt |
Surprising findings
- Conventional proxy-reward ILfO agents on the basketball task learn to sweep the ball away and fail to grasp it, even though grasping alone is learned quickly when only early demonstration steps are used.
- Rewarding later steps of the expert trajectory actively hinders learning of earlier behaviors in progress-dependent tasks.
Limitations
- Evaluated only on Meta-World manipulation tasks with simulated demonstrations from hard-coded policies.
- Relies on hyperparameters lambda and alpha for progress recognizer and mapping function, though claimed robust.
- Model-based components are not integrated; authors leave this to future work.
Experimental setup
Datasets: Meta-World expert demonstrations generated by hard-coded policies (10 per task)
Benchmarks: Meta-World basketball assembly hammer door unlock lever pull stick push door lock pick place wall door open button press
Metrics: task success rate sample efficiency (environment frames)
Baselines: OT GAIfO BCO RL with goal-based rewards RL with LIS rewards exponential discount scheduling
Builds on: optimal transport imitation (OT) GAIfO DrQ-v2 BCO Meta-World
Closest papers by meaning
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.90)
- RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Causal Imitation for Markov Decision Processes: a Partial Identification Approach (NeurIPS 2024 · similarity 0.90)
- Efficient Imitation under Misspecification (ICLR 2025 · similarity 0.89)
- Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment (NeurIPS 2024 · similarity 0.89)
- GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.