Research map / Reinforcement learning / Imitation learning
Diverse Policies Recovering via Pointwise Mutual Information Weighted Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BC-PMI, a style-conditioned behavioral cloning objective weighted by pointwise mutual information (estimated with MINE) so that state-action pairs relevant to a trajectory's latent style get more weight.
Problem
Recovering a spectrum of diverse policies from expert trajectories with known style labels, where vanilla style-conditioned behavioral cloning treats all state-action pairs equally even though style relevance varies across a trajectory.
Approach
Weight each state-action pair's style-conditioned BC loss by exp(PMI(s,a;z)) = p(z|s,a)/p(z), with PMI estimated by a MINE network and a moving-average baseline subtracted to reduce gradient variance; theory shows this interpolates between vanilla BC (zero MI) and per-style clustered BC (zero conditional entropy).
Key results
- On Atari, BC-PMI reaches 0.61 position-style calibration on Alien and 0.83 on SpaceInvaders, versus 0.52 and 0.66 for the best baseline (CTVAE).
- On Circle 2D, BC-PMI lowers Class 3 DTW from 18.97 (CBC) to 11.58.
- On basketball Destination style, BC-PMI reaches 93.3% / 92.7% / 89.8% class calibration versus at most 84.4% for baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PMI weighting to style-conditioned behavioral cloning | BC-PMI vs CBC · Circle 2D, Class 1 | DTW: 8.130 $\pm$ 0.732 -> 7.511 $\pm$ 0.545 | Helped |
| Add PMI weighting to style-conditioned behavioral cloning | BC-PMI vs CBC · Circle 2D, Class 3 | ED: 107.833 $\pm$ 8.213 -> 74.267 $\pm$ 8.002 | Helped |
| Add PMI weighting to style-conditioned behavioral cloning | BC-PMI vs strongest baseline · Alien, Pos | style calibration: 0.52$\pm$0.04 -> 0.61$\pm$0.08 | Helped |
| Add PMI weighting to style-conditioned behavioral cloning | BC-PMI vs strongest baseline · Alien, Range | style calibration: 0.57$\pm$0.09 -> 0.55$\pm$0.12 | Hurt |
| Add PMI weighting to style-conditioned behavioral cloning | BC-PMI vs strongest baseline · MsPacman, Pos | style calibration: 0.55$\pm$0.04 -> 0.57$\pm$0.07 | Helped |
| Add PMI weighting to style-conditioned behavioral cloning | BC-PMI vs strongest baseline · MsPacman, Range | style calibration: 0.61$\pm$0.05 -> 0.56$\pm$0.14 | Hurt |
| Add PMI weighting to style-conditioned behavioral cloning | BC-PMI vs strongest baseline · Space Invaders, Pos and Fire | style calibration: 0.71$\pm$0.08 -> 0.83$\pm$0.06; 0.47$\pm$0.06 -> 0.51$\pm$0.10 | Helped |
| Add PMI weighting to style-conditioned behavioral cloning | BC-PMI vs CBC · Basketball, Destination Class 1 and Class 2 | style calibration (%): 81.6 -> 93.3; 84.4 -> 92.7 | Helped |
| Add PMI weighting to style-conditioned behavioral cloning | BC-PMI vs CBC · Basketball, Curvature Classes 1–3 | style calibration (%): 67.3 -> 79.4; 66.2 -> 80.9; 71.6 -> 81.6 | Helped |
| Use styles related to position or range rather than an unconditioned BC policy | BC-PMI · Alien, cumulative reward | cumulative reward: 460.5$\pm$173.4; 416.0$\pm$160.8; 432.0$\pm$174.7; 500.3$\pm$211.7; 496.1$\pm$191.5 | Mixed |
| Use high firing frequency rather than low firing frequency | BC-PMI Space Invaders Fire styles · Space Invaders | cumulative reward: 94.8$\pm$47.2 -> 270.6$\pm$86.7 | Helped |
Surprising findings
- On range-type styles BC-PMI is only comparable to CTVAE rather than best, which the authors attribute to weaker style-distinguishing mutual information for those styles.
- In Space Invaders, the BC-PMI policy conditioned on the low fire-rate style earns a lower cumulative reward (94.8) than BC (208.2), showing that some styles strongly affect game score.
Limitations
- Assumes the expert offline data already meets performance requirements; cannot improve policy performance beyond the demonstrations.
- Requires style labels to be provided (via labeling functions or inference), so it does not address unsupervised style discovery.
- Weaker gains for range-type styles where the style is less distinguishable by mutual information.
Experimental setup
Datasets: Circle 2D (synthetic) Atari-HEAD NBA professional basketball player tracking data
Benchmarks: Circle 2D Atari Alien Atari MsPacman Atari SpaceInvaders NBA basketball player trajectories
Metrics: DTW ED KL divergence style calibration accuracy cumulative reward mutual information lower bound
Baselines: BC CBC CTVAE InfoGAIL SORL
Builds on: Mutual Information Neural Estimation (MINE) behavioral cloning InfoGAIL SORL A3C baseline
Closest papers by meaning
- How to Leverage Diverse Demonstrations in Offline Imitation Learning (ICML 2024 · similarity 0.89)
- Split the Differences, Pool the Rest: Provably Efficient Multi-Objective Imitation (NeurIPS 2026 · similarity 0.88)
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.88)
- Learning When to Stop: Selective Imitation Learning Under Arbitrary Dynamics Shift (NeurIPS 2026 · similarity 0.87)
- Contractive Dynamical Imitation Policies for Efficient Out-of-Sample Recovery (ICLR 2025 · similarity 0.87)
- Rethinking Latent Redundancy in Behavior Cloning: An Information Bottleneck Approach for Robot Manipulation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.