Research map / Reinforcement learning / Imitation learning
Mimicking Better by Matching the Approximate Action Distribution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MAAD, an on-policy imitation-from-observations method that adds a KL regularizer aligning the policy with an inverse dynamics model's action distribution, applied to adversarial, trajectory-matching, and optimal-transport surrogate rewards on MuJoCo tasks.
Problem
Imitation learning from observations (no expert actions) is sample-inefficient for on-policy methods; behavioral-cloning-style regularization needs actions that are unavailable.
Approach
Learn a mixture-density inverse dynamics model from the agent's replay buffer to infer plausible actions for expert state-state transitions, then regularize the on-policy PPO imitator with a forward KL between the inverse model and the policy, usable with any surrogate reward (AIL, trajectory matching, OT).
Key results
- MAAD-AIL reaches expert performance in both suites and nearly all environments, except dog-run in DMC, with up to four times greater sample efficiency than GAIfO on Gym.
- Non-regularized TMO and OTO fail to converge within the interaction budget on DMC, while MAAD-TM and MAAD-OT converge on most tasks.
- MAAD-AIL slightly outperforms GAIL-BC, which has access to expert actions, on several Gym tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add inverse-dynamics action-distribution regularization to the adversarial imitation method (ablation) | GAIfO / MAAD-AIL · MuJoCo tasks, OpenAI Gym and DMC Suite | sample efficiency | Helped |
| Add inverse-dynamics action-distribution regularization to trajectory matching (ablation) | TMO / MAAD-TM · DMC Suite | sample efficiency | Helped |
| Add inverse-dynamics action-distribution regularization to optimal transport (ablation) | OTO / MAAD-OT · DMC Suite | sample efficiency | Helped |
| Use MAAD-TM or MAAD-OT rather than their non-regularized variants (ablation) | TMO and OTO · OpenAI Gym | sample efficiency | Mixed |
| Use MAAD-AIL instead of GAIfO | Adversarial imitation learning · OpenAI Gym | sample efficiency | Helped |
| Increase inverse-dynamics mixture components above K=1 (ablation) | MAAD inverse dynamics model · MuJoCo suites | performance | No clear effect |
Surprising findings
- MAAD-AIL, which uses no expert actions, slightly outperforms GAIL-BC, which has access to expert actions.
- TM and OT variants fail on walker-based tasks, with negative R-squared against expert actions in some cases.
Limitations
- Walker-based environments are poorly learned by TM and OT variants, with negative R-squared for some action fits.
- Inverse model assumed effectively unimodal (K=1) based on preliminary sweeps; multimodality not exploited.
- Experiments limited to MuJoCo locomotion with 3 seeds and expert data from a PPO policy.
Experimental setup
Datasets: MuJoCo expert trajectories generated by PPO on OpenAI Gym and DMC tasks
Benchmarks: Hopper-v3 HalfCheetah-v3 Walker2d-v3 Ant-v3 Humanoid-v3 DMC Cheetah-Run DMC Walker-Walk DMC Walker-Run DMC Quadruped-Run DMC Quadruped-Walk DMC Dog-Run
Metrics: environment interactions to expert performance episode return R-squared between learned and expert actions
Baselines: GAIL-BC GAIfO TMO OTO BC
Builds on: GAIL GAIfO PPO ROT TD3-BC
Closest papers by meaning
- Split the Differences, Pool the Rest: Provably Efficient Multi-Objective Imitation (NeurIPS 2026 · similarity 0.88)
- Efficient Active Imitation Learning with Random Network Distillation (ICLR 2025 · similarity 0.87)
- Diffusion Imitation from Observation (NeurIPS 2024 · similarity 0.87)
- Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration (ICLR 2025 · similarity 0.87)
- Efficient Imitation under Misspecification (ICLR 2025 · similarity 0.87)
- Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.