Research map / Reinforcement learning / Imitation learning
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, Scott Niekum
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn a reward from ranked demonstrations, then optimize it with reinforcement learning instead of directly imitating demonstrations | T-REX · Atari (7 of 8 games) | score versus best demonstration | Improved |
| Learn a reward from ranked demonstrations, then optimize it with reinforcement learning instead of imitation | T-REX · HalfCheetah and Hopper, all stages | forward distance versus best demonstration | Improved |
| Use time order as a proxy for preference rankings | T-REX · MuJoCo tasks and stages | forward distance versus BCO and GAIL | Mixed |
| Use noisy human rankings instead of ground-truth trajectory rankings | T-REX · Atari (5 of 8 games) | score versus best demonstration | Mixed |
| Rank demonstrations by human preference | T-REX · Q*bert human demonstrations | score versus best human demonstration: 875 -> 6,869.2 | Improved |
| Rank demonstrations by human preference | T-REX · Montezuma's Revenge human demonstrations | score versus best human demonstration: 2,600 -> 0.0 | Hurt |
| Compare partial trajectories with time-aligned starting points rather than randomly chosen partial trajectories (ablation) | Atari T-REX reward training · Atari games other than Enduro | policy performance | Improved |
| Train on short partial trajectories rather than down-sampled full trajectories (ablation) | Enduro T-REX reward training · Enduro | score | Hurt |
| Infer reward from rankings with a neural network ensemble | T-REX · Pong | score: -6 -> -2.0 | Mixed |
| Infer reward from rankings with a neural network ensemble | T-REX · Hero | score: 13,235 -> 0 | Hurt |
| Use ground-truth trajectory rankings rather than timestamp-based rankings (ablation) | T-REX · Ant Stage 2 | forward distance: 1.28 (0.28) -> 7.34 (2.50) | Improved |
| Use timestamp-based rankings rather than ground-truth rankings (ablation) | T-REX · HalfCheetah Stage 1 | forward distance: 46.90 (1.89) -> 51.39(4.52) | Improved |
Closest papers by meaning
- Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration (ICLR 2025 · similarity 0.90)
- Environment Design for Inverse Reinforcement Learning (ICML 2024 · similarity 0.88)
- Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.88)
- Inverse Reinforcement Learning by Estimating Expertise of Demonstrators (AAAI 2025 · similarity 0.88)
- Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment (NeurIPS 2024 · similarity 0.88)
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
