AI Paper Atlas

Research map / Reinforcement learning / Imitation learning

Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations

Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, Scott Niekum

ICML 2019 · Reinforcement learning › Imitation learning · approach: Reward learning from demonstrations · 30 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Learn a reward from ranked demonstrations, then optimize it with reinforcement learning instead of directly imitating demonstrationsT-REX · Atari (7 of 8 games)score versus best demonstrationImproved
Learn a reward from ranked demonstrations, then optimize it with reinforcement learning instead of imitationT-REX · HalfCheetah and Hopper, all stagesforward distance versus best demonstrationImproved
Use time order as a proxy for preference rankingsT-REX · MuJoCo tasks and stagesforward distance versus BCO and GAILMixed
Use noisy human rankings instead of ground-truth trajectory rankingsT-REX · Atari (5 of 8 games)score versus best demonstrationMixed
Rank demonstrations by human preferenceT-REX · Q*bert human demonstrationsscore versus best human demonstration: 875 -> 6,869.2Improved
Rank demonstrations by human preferenceT-REX · Montezuma's Revenge human demonstrationsscore versus best human demonstration: 2,600 -> 0.0Hurt
Compare partial trajectories with time-aligned starting points rather than randomly chosen partial trajectories (ablation)Atari T-REX reward training · Atari games other than Enduropolicy performanceImproved
Train on short partial trajectories rather than down-sampled full trajectories (ablation)Enduro T-REX reward training · EnduroscoreHurt
Infer reward from rankings with a neural network ensembleT-REX · Pongscore: -6 -> -2.0Mixed
Infer reward from rankings with a neural network ensembleT-REX · Heroscore: 13,235 -> 0Hurt
Use ground-truth trajectory rankings rather than timestamp-based rankings (ablation)T-REX · Ant Stage 2forward distance: 1.28 (0.28) -> 7.34 (2.50)Improved
Use timestamp-based rankings rather than ground-truth rankings (ablation)T-REX · HalfCheetah Stage 1forward distance: 46.90 (1.89) -> 51.39(4.52)Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy