Research map / Reinforcement learning / Imitation learning
SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPRINQL, a non-adversarial inverse soft-Q learning method for offline imitation learning that uses expert plus multiple tiers of sub-optimal demonstrations, with a reference-reward regularizer and preference-learned tier weights.
Problem
Offline imitation learning when expert demonstrations are scarce and only a larger set of sub-optimal demonstrations of varying quality is available; existing BC and distribution-matching methods overfit to few experts or imitate sub-optimal data.
Approach
Extends inverse soft-Q learning to multiple expertise levels with learned weights per level, adds a reward-regularization term toward a reference reward inferred from ranked demonstrations via a Bradley-Terry preference loss, and makes the objective concave in Q (with a lower-bound surrogate) so training is a non-adversarial maximization over Q.
Key results
- With three expertise levels, SPRINQL averages 67.7 across three Mujoco and three Panda-gym tasks, versus 14.6 for the best baseline (DWBC) on the same table.
- On Mujoco HalfCheetah, SPRINQL scores 73.6 versus 14.2 for the best BC baseline (BC-O).
- Full Mujoco average is 78.3 for SPRINQL versus 31.9 for W-BC; Panda-gym average is 68.2 versus 41.7 for DWBC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SPRINQL instead of BC-O | Offline imitation learning · Cheetah | normalized return: 14.2$\pm$2.9 -> 73.6$\pm$4.3 | Helped |
| Use SPRINQL instead of W-BC | Offline imitation learning · Ant | normalized return: 47.3$\pm$6.4 -> 77.0$\pm$5.6 | Helped |
| Use SPRINQL instead of W-BC | Offline imitation learning · Humanoid | normalized return: 19.6$\pm$19.0 -> 82.9$\pm$11.2 | Helped |
| Use SPRINQL instead of DWBC | Offline imitation learning · Push | normalized return: 36.9$\pm$7.4 -> 72.0$\pm$5.3 | Helped |
| Use SPRINQL instead of DWBC | Offline imitation learning · PnP | normalized return: 25.0$\pm$6.3 -> 63.2$\pm$6.4 | Helped |
| Use SPRINQL instead of DWBC | Offline imitation learning · Slide | normalized return: 11.6$\pm$4.4 -> 37.7$\pm$6.6 | Helped |
| Add the reward regularizer (ablation) | SPRINQL · Mujoco tasks, N=2 and N=3 | normalized return | Helped |
| Add distribution matching (ablation) | SPRINQL · Mujoco tasks, N=2 and N=3 | normalized return | Helped |
| Add the conservative term (ablation) | SPRINQL · Offline imitation learning benchmarks | normalized return | Helped |
Surprising findings
- Naively mixing all sub-optimal data with expert data (BC-both, IQ-both, SQIL-both) often performs worse than using expert data alone or is near zero on several Mujoco tasks.
Limitations
- Requires a large amount of sub-optimal data with well-identified expertise levels.
- Lacks theoretical analysis of how expert and non-expert dataset sizes affect Q-learning performance.
- No theoretical explanation of how the reward regularizer helps distribution matching when expert samples are scarce.
Experimental setup
Datasets: Mujoco expert and sub-optimal demonstrations (generated) Panda-gym expert and sub-optimal demonstrations (generated)
Benchmarks: Mujoco HalfCheetah Mujoco Ant Mujoco Humanoid Mujoco Walker Mujoco Hopper Panda-gym Reach Panda-gym Push Panda-gym PickAndPlace Panda-gym Slide
Metrics: normalized return score (0-100)
Baselines: BC-E BC-O BC-both W-BC TRAIL IQ-Learn SQIL DemoDICE DWBC
Builds on: IQ-Learn T-REX Conservative Q-learning MaxEnt IRL SQIL
Closest papers by meaning
- How to Leverage Diverse Demonstrations in Offline Imitation Learning (ICML 2024 · similarity 0.89)
- SEABO: A Simple Search-Based Method for Offline Imitation Learning (ICLR 2024 · similarity 0.89)
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning (NeurIPS 2024 · similarity 0.89)
- GO-DICE: Goal-Conditioned Option-Aware Offline Imitation Learning via Stationary Distribution Correction Estimation (AAAI 2024 · similarity 0.88)
- Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms (ICML 2024 · similarity 0.88)
- Inverse Reinforcement Learning by Estimating Expertise of Demonstrators (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.