Research map / Reinforcement learning / Imitation learning
Markov Balance Satisfaction Improves Performance in Strictly Batch Offline Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MBIL, a strictly batch offline imitation learning method that enforces the Markov balance equation using conditional normalizing flows (RealNVP/GLOW-style) for transition density estimation, combined with an MSE behavior cloning policy loss.
Problem
Imitation learning from a single expert dataset with no environment interaction, no supplementary data, and no access to transition dynamics; prior DICE/IRL methods suffer termination/reward bias and need on-policy samples or extra data.
Approach
Replace reward estimation and stationary distribution matching with a dynamics loss that enforces the Markov balance equation P(s',a'|s,a)=pi(a'|s')T(s'|s,a), where both conditional densities are estimated by conditional normalizing flows, regularized with a behavior cloning policy loss.
Key results
- On classic control, MBIL reaches near-expert performance with at most three trajectories and near-expert on CartPole with a single trajectory, generally outperforming BC, ValueDICE, RCAL, EDM, AVRIL, DSFN and IQ-Learn.
- On MuJoCo with one expert trajectory, MBIL consistently achieves near-expert performance across Hopper, Ant, HalfCheetah and Walker2d, while BC degrades on Ant and Walker2d with more gradient updates.
- Ablation on Walker2d shows that combining policy and dynamics losses, with higher weight on the policy loss, is needed to reach near-expert performance from one trajectory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MBIL instead of offline IL/IRL/AIL baselines | Strictly batch imitation learning · Acrobot, CartPole, LunarLander | average reward | Helped |
| Use MBIL instead of BC, ValueDICE, SoftDICE, and ODICE | Strictly batch imitation learning · Hopper, Ant, HalfCheetah, Walker2d | average reward | Helped |
| Combine policy loss and dynamics loss, with higher weight on policy loss (ablation) | MBIL · Walker2d | average reward | Helped |
| Add dynamics loss to policy loss, using a higher relative weight on policy loss (ablation) | MBIL · Walker2d | average reward | Helped |
| Use dynamics loss alone instead of policy loss alone (ablation) | MBIL · Walker2d | average reward | Helped |
Surprising findings
- Dynamics loss alone and policy (BC) loss alone perform similarly on Walker2d, and only their combination reaches near-expert performance.
Limitations
- Assumes expert (optimal) demonstrations; extensions to suboptimal data are left to future work.
- No sample complexity bounds are derived.
- Excludes methods requiring additional data or environment interaction (e.g., CLARE, DemoDICE, MILO, OPOLO) from comparison.
Experimental setup
Datasets: RL Baselines Zoo expert trajectories (Acrobot-v1, CartPole-v1, LunarLander-v2) GAIL-generated expert data for MuJoCo (Hopper, Ant, HalfCheetah, Walker2d)
Benchmarks: Acrobot-v1 CartPole-v1 LunarLander-v2 Hopper Ant HalfCheetah Walker2d
Metrics: average episode reward
Baselines: BC ValueDICE RCAL EDM AVRIL DSFN IQ-Learn SoftDICE ODICE
Builds on: ValueDICE SoftDICE ODICE RealNVP Glow GAIL
Closest papers by meaning
- Reward-free World Models for Online Imitation Learning (ICML 2025 · similarity 0.89)
- How to Leverage Diverse Demonstrations in Offline Imitation Learning (ICML 2024 · similarity 0.89)
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.89)
- Imitation from Observations with Trajectory-Level Generative Embeddings (NeurIPS 2026 · similarity 0.89)
- Causal Imitation for Markov Decision Processes: a Partial Identification Approach (NeurIPS 2024 · similarity 0.88)
- CCIL: Continuity-Based Data Augmentation for Corrective Imitation Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.