Research map / Reinforcement learning / Imitation learning

Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning

AAAI 2024 · Reinforcement learning › Imitation learning · approach: Imitation combined with RL · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SERLfD, which uses a self-explanation network (SE-Net) inside a GAN-style discriminator to predict predicate utility weights that augment states and rewards for RL from demonstrations, improving stability and performance on robot pushing and Pacman tasks.

Problem

Reinforcement learning from demonstrations becomes sample-inefficient and unstable when demonstrations are ambiguous (multiple valid interpretations of the same demo), and an optimal demo can also be ambiguous.

Approach

An SE-Net predicts per-predicate utility weights from raw states, trained as part of a discriminator that separates successful from failed experiences (no self-explanation labels). The weights are used to augment RL states with predicate values and utilities and to form potential-based shaping rewards, so the agent learns which relations matter for the task while maximizing environment reward.

Key results

• SE-augmented TD3fD and SACfD (TD3fD/SACfD+SE) show increased stability and higher scores in a shorter timeframe than baselines in the robot push domains.
• In Robot-Push-Simple with SACfD, self-explanations matter less because the task is simpler and entropy-driven exploration suffices.
• SQLfD+SE outperforms SQLfD in the discrete Pacman domain, per the paper's reported results.

Surprising findings

• Self-explanation benefits are smaller in Robot-Push-Simple with SACfD, attributed to entropy-driven exploration.
• Using -1 for unsatisfied predicates worked better than 0 empirically.

Limitations

• Relies on optimal object trackers and hand-built predicate classifiers; real-world noise not evaluated.
• Evaluated only in simulation (PyBullet) and a Pacman domain; no real robot experiments.
• High standard deviation across runs in some ablations; only three runs per curve.

Experimental setup

Datasets: human keyboard demonstrations in robot push domains Pacman demonstrations

Benchmarks: Robot-Push-Simple Robot-Push Robot-Remove-and-Push Pacman

Metrics: score (average over 100 episodes) training stability (max/min/std across 3 runs)

Baselines: TD3fD SACfD SA-GAN-GCL SQLfD

Builds on: TD3fD SACfD Guided Cost Learning Adversarial IRL Soft Q-Learning from demonstrations

Closest papers by meaning

• RLfOLD: Reinforcement Learning from Online Demonstrations in Urban Autonomous Driving (AAAI 2024 · similarity 0.87)
• ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
• Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.85)
• Grounding Language Plans in Demonstrations Through Counterfactual Perturbations (ICLR 2024 · similarity 0.85)
• Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data (ICLR 2024 · similarity 0.85)
• RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR 2024 · similarity 0.85)