Research map / Reinforcement learning / Imitation learning
IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SOAR, a template that makes SAC-based imitation learning optimistic by using an ensemble of critics (mean minus clipped std), with tabular regret guarantees and boosts to f-IRL, ML-IRL, CSIL and RKL on MuJoCo.
Problem
Imitation learning built on Soft Actor Critic lacks efficient exploration; prior theoretically grounded exploration bonuses only work in tabular or linear settings and cannot be implemented with neural networks.
Approach
Replace the single SAC critic pair with L critics trained on independent data, and use an optimistic estimate (ensemble mean minus clipped standard deviation) in the SAC policy update, within a primal-dual cost/policy alternation; tabular version has provable sample complexity.
Key results
- SOAR-augmented SAC-based IL methods (f-IRL, ML-IRL, CSIL, RKL) consistently outperform their base versions across MuJoCo environments.
- SOAR reaches the best baseline performance with roughly half the number of learning episodes.
- Tabular SOAR-IL achieves O(epsilon^-2) expert trajectories and interactions, matching the best known rates in the desired accuracy parameter.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the SAC single critic with multiple critics and use an optimistic critic estimate | SAC-based imitation learning algorithms, including RKL, ML-IRL, and CSIL · MuJoCo environments; average normalized return across 4 environments with 16 expert trajectories | average normalized return | Helped |
| Use SOAR-enhanced methods rather than the base methods | SAC-based imitation learning algorithms · Learning episodes to achieve the same performance | episodes: reduced by half | Helped |
| Use 1 critic network instead of multiple critics (ablation) | SOAR-IL · Two-state hard-exploration MDP | return | Hurt |
| Use 2 critic networks rather than 1 (ablation) | SOAR-IL · Two-state hard-exploration MDP | task performance | Helped |
| Increase the critic count from 2 to 3 (ablation) | SOAR-IL · Two-state hard-exploration MDP | variance | Helped |
| Increase the number of critics further beyond 3 (ablation) | SOAR-IL · Two-state hard-exploration MDP | MDP samples needed to solve | Hurt |
| Use L=4 critics and an appropriately chosen standard-deviation clipping threshold (ablation) | RKL and ML-IRL (state-only) · MuJoCo environments | performance | Helped |
Surprising findings
- With a single critic (L=1), the hard two-state exploration task is not solved reliably; high variance across seeds, and larger L beyond a point degrades sample efficiency.
Limitations
- Theoretical guarantees are restricted to the tabular setting; the neural version lacks a formal optimism guarantee.
- Number of critics L and std clipping threshold sigma require tuning per algorithm.
Experimental setup
Datasets: MuJoCo expert demonstrations generated by SAC policies
Benchmarks: MuJoCo Ant-v5 Hopper-v5 Humanoid-v5 Walker2d-v5 two-state hard exploration MDP
Metrics: normalized return required number of episodes
Baselines: CSIL ML-IRL RKL f-IRL OPT-AIL GAIL SQIL
Builds on: Soft Actor Critic CSIL ML-IRL RKL f-IRL
Closest papers by meaning
- Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration (ICLR 2025 · similarity 0.89)
- Agnostic Interactive Imitation Learning: New Theory and Practical Algorithms (ICML 2024 · similarity 0.87)
- Imitation Learning from Observation with Automatic Discount Scheduling (ICLR 2024 · similarity 0.87)
- Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning (NeurIPS 2025 · similarity 0.87)
- Imitation Learning in Discounted Linear MDPs without exploration assumptions (ICML 2024 · similarity 0.87)
- Offline Actor-Critic Reinforcement Learning Scales to Large Models (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.