Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Deep Implicit Imitation Reinforcement Learning in Heterogeneous Action Settings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends deep implicit imitation Q-networks (DIIQN) to heterogeneous action sets between mentor and observer, using a deep k-n step repair bridging mechanism to find alternative paths into mentor trajectories.
Problem
Implicit imitation RL assumes mentor and trainee share the same action space; in real settings physical differences give the mentor distinct actions, so state-only imitation breaks down.
Approach
HA-DIIQN flags mentor transitions as infeasible using a KL-based running error, searches the observer's replay buffer for short bridges (deep k-n step repair) back into the mentor trajectory, and replaces the mentor loss target with the bridge's first action and next state, with no environment simulation or brute-force search.
Key results
- In the 2D maze, HA-DIIQN converged at about episode 1140 versus about 3760 for DIIQN and about 3900 for DQN (averaged over 30 runs).
- In Point Maze, HA-DIIQN converged at about episode 110, compared with about 250 for DIIQN and 260 for DQN.
- DIIQN gave only marginal speed-up over DQN in the maze because mentor actions were largely irrelevant to the observer's action set.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HA-DIIQN instead of DQN with a heterogeneous mentor dataset | 2D Maze 30×30 trainee · 2D Maze 30×30 | episodes to convergence: episode 3900; episode 1140 | Helped |
| Use HA-DIIQN instead of DIIQN with a heterogeneous mentor dataset | 2D Maze 30×30 trainee · 2D Maze 30×30 | episodes to convergence: approximately episode 3760; episode 1140 | Helped |
| Use HA-DIIQN instead of DQN with a heterogeneous mentor dataset | Point Maze trainee · Point Maze | episodes to convergence: 260 episodes; episode 110 | Helped |
| Use HA-DIIQN instead of DIIQN with a heterogeneous mentor dataset | Point Maze trainee · Point Maze | episodes to convergence: 250 episode mark; episode 110 | Helped |
| Use DIIQN instead of DQN with heterogeneous mentorship | 2D Maze 30×30 trainee · 2D Maze 30×30 | episodes to convergence: episode 3900; approximately episode 3760 | Helped |
| Use DIIQN instead of DQN with heterogeneous mentorship | Point Maze trainee · Point Maze | episodes to convergence: 260 episodes; 250 episode mark | No clear effect |
Surprising findings
- Standard DIIQN gives little or no benefit over DQN when the mentor's action set differs, and in Point Maze it converges at roughly the same time as DQN.
Limitations
- Bridge search relies on replay buffer contents, so overwritten transitions can limit bridge discovery and the method depends on observer exploration.
- Requires empirically set thresholds (e_infeas, e_th) and search depths k and n, which are domain-specific.
- Evaluated only on a 2D maze and a single Point Maze setting, with a mentor restricted to a single demonstration in the maze.
Experimental setup
Datasets: OpenAI Gym 2D Maze (30x30) D4RL Point Maze
Benchmarks: 2D Maze 30x30 (OpenAI Gym) D4RL Point Maze
Metrics: reward per episode episodes to convergence
Baselines: DQN DIIQN
Builds on: DIIQN DQN implicit imitation RL (Price and Boutilier)
Closest papers by meaning
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.87)
- Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.87)
- Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR 2024 · similarity 0.86)
- Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.86)
- Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.