Research map / Reinforcement learning / Imitation learning
Action-Constrained Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Action-Constrained Imitation Learning (ACIL) and DTWIL, which uses MPC with a DTW state-sequence alignment to generate feasible surrogate expert demonstrations, then runs BC on them.
Problem
Imitating an unconstrained expert with an imitator whose action space is restricted by constraints causes occupancy measure distortion, so standard IL and projection-based methods fail to match expert state trajectories.
Approach
Recast trajectory alignment as MPC planning that generates action-feasible surrogate trajectories matching expert state sequences under a DTW distance (handling different trajectory lengths), then apply any off-the-shelf IL method (e.g., BC) to these surrogate demonstrations.
Key results
- On Hopper H+B, DTWIL reaches 2844.68 ± 57.77 return vs. 1375 ± 790 for SAIL and 1068 ± 952 for OPOLO under a 50K environment-step budget.
- On HalfCheetah HC+B, DTWIL achieves 2669 ± 4 return vs. 1222 ± 260 for SAIL.
- On Maze2d M+B, DTWIL gets 0.77 ± 0.04 return vs. 0.45 ± 0.06 for SAIL; on Table-Wiping W+L DTWIL reaches 70 ± 4 vs. 91 ± 17 for BCO, so BCO is better there.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace L2 alignment with DTW alignment (ablation) | DTWIL · HalfCheetah HC+B, HC+O; Hopper H+B, H+M | return: 2157.52 ± 60.84 -> 2669.41 ± 4.56; 2279.12 ± 51.18 -> 2637.34 ± 26.82; 1054.49 ± 227.64 -> 2844.68 ± 57.77; 1245.61 ± 47 | Helped |
| add DTW input normalization (ablation) | DTWIL · HalfCheetah HC+B and Hopper H+B | Return-BC: 1893.90 ± 71.56 -> 2669.41 ± 4.56; 281.13 ± 31.88 -> 2844.68 ± 57.77 | Helped |
| use asynchronous instead of synchronous progression (ablation) | DTWIL · Hopper H+B | Return-BC: 153.52 ± 1.20 -> 2844.68 ± 57.77 | Helped |
| use asynchronous instead of synchronous progression (ablation) | DTWIL · HalfCheetah HC+B | Return-BC: 2594.28 ± 29.80 -> 2669.41 ± 4.56 | No clear effect |
| add Expert Regularized Control (ablation) | DTWIL · Hopper | Return-BC: 889.7 ± 5.4 -> 2844.7 ± 57.8 | Helped |
| set ERC beta to 0.05 rather than 0, 0.02, 0.1, or 0.2 (ablation) | DTWIL · Hopper H+M | Return-BC: 889.65 ± 5.39; 1138.85 ± 56.35; 2167.30 ± 360.73; 723.95 ± 345.70 -> 2844.68 ± 57.77 | Helped |
| set ERC horizon to 5 rather than 0, 10, or 20 (ablation) | DTWIL · Hopper H+M | Return-BC: 889.65 ± 5.39; 2686.85 ± 135.64; 2616.09 ± 102.90 -> 2844.68 ± 57.77 | Helped |
| replan after every step rather than executing 3 or 5 steps (ablation) | DTWIL · HalfCheetah HC+B and Hopper H+B | return: 2427 ± 0 -> 2669 ± 4; 1270 ± 464 -> 2669 ± 4; 810 ± 21 -> 2844 ± 57; 579 ± 2 -> 2844 ± 57 | Helped |
| use DTWIL-generated surrogate data with BC instead of direct BC (ablation) | BC · M+B, M+O, H+B, H+M | return: 0.61 ± 0.05 -> 0.77 ± 0.04; 0.81 ± 0.05 -> 0.87 ± 0.04; 2204.83 ± 753.32 -> 2844.68 ± 57.77; 1233.96 ± 211.87 -> 2873.88 | Helped |
Surprising findings
- BCO outperforms DTWIL on the Table-Wiping W+B and W+L tasks (BCO 59 and 91 vs. DTWIL 61 and 70 for W+B and W+L respectively is mixed; BCO is higher on W+L).
- Re-planning every 3 or 5 steps instead of every step substantially degrades performance, e.g., Hopper H+B drops from 2844 to 810 at 3 steps.
Limitations
- Re-planning at every step is computationally expensive, and multi-step execution degrades alignment quality.
- Relies on a learned dynamics model trained largely on expert trajectories, with online augmentation needed to avoid overfitting.
Experimental setup
Datasets: Maze2d-Medium-v1 expert demos HalfCheetah expert demos Hopper expert demos Robosuite Table-Wiping expert demos
Benchmarks: Maze2d-Medium-v1 HalfCheetah-v3 Hopper-v2 Robosuite Table-Wiping
Metrics: episode return DTW distance to expert trajectory
Baselines: BCO GAIL GAIfO OPOLO SAIL CFIL-s CFIL-sa DIFO BC
Builds on: PETS Actor Regularized Control GAIL CFIL OPOLO
Closest papers by meaning
- DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning (NeurIPS 2024 · similarity 0.88)
- Split the Differences, Pool the Rest: Provably Efficient Multi-Objective Imitation (NeurIPS 2026 · similarity 0.87)
- Implicit Drifting Policy: One-Step Action Generation via Conditional Expert Geometry (NeurIPS 2026 · similarity 0.87)
- Simplifying Constraint Inference with Inverse Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learning (ICML 2024 · similarity 0.87)
- Limited Preference Aided Imitation Learning from Imperfect Demonstrations (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.