Research map / Reinforcement learning / Imitation learning
Making AI Policies Transparent to Humans through Demonstrations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes teaching AI reward functions to humans via demonstrations chosen in the zone of proximal development, using a particle-filter human model and closed-loop scaffolded teaching with tests.
Problem
Making AI policies transparent to humans: humans learning reward functions from demonstrations are computationally limited, so demonstrations that maximize information gain may be too hard to understand.
Approach
Select demonstrations that balance informativeness and difficulty (zone of proximal development) by modeling human beliefs as a particle filter over reward functions, and scaffold them in a closed-loop teaching algorithm with diagnostic tests, feedback, and remedial demonstrations.
Key results
- Closed-loop algorithm reduces regret in human test responses by 41% over a baseline (under review study, per paper).
- Scaffolded demonstration curriculum improved performance on tests of later-demonstrated concepts but decreased performance on early-demonstrated concepts in a second user study.
- Information gain often correlated with the effort required for learners to process demonstrations in a first user study.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use scaffolded demonstrations that incrementally increase in information | Teaching a robot's reward function · Tests of later-demonstrated reward concepts | performance | Helped |
| Use scaffolded demonstrations that incrementally increase in information | Teaching a robot's reward function · Tests of early-demonstrated reward concepts | performance | Hurt |
| Use the closed-loop teaching algorithm with diagnostic tests, feedback, and remedial demonstrations | Teaching related knowledge concepts · Human test responses | regret: 41% | Helped |
| Use the closed-loop teaching algorithm | Teaching a robot's reward function · One of the two considered domains | usability | Helped |
Surprising findings
- Scaffolded demonstrations decreased participants' performance on tests of early-demonstrated concepts, suggesting challenge without feedback can hurt early learning.
Limitations
- Work is a research statement summarizing prior and ongoing studies; the closed-loop results are under review and not detailed.
- Current experiments are in grid-world environments.
- High-dimensional reward functions and complex domains remain unaddressed.
Experimental setup
Metrics: regret in human test responses usability ratings test performance on later- and early-demonstrated concepts
Baselines: information-gain demonstration selection (Lage et al. 2019) Huang et al. 2019 Qian and Unhelkar 2022
Builds on: inverse reinforcement learning zone of proximal development knowledge component framework
Closest papers by meaning
- Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning (ICLR 2021 · similarity 0.87)
- Showing versus doing: Teaching by demonstration (NeurIPS 2016 · similarity 0.87)
- When a Robot is More Capable than a Human: Learning from Constrained Demonstrators (ICLR 2026 · similarity 0.86)
- Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale (NeurIPS 2024 · similarity 0.86)
- Imitation Learning from Observation with Automatic Discount Scheduling (ICLR 2024 · similarity 0.85)
- Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.