Research map / Reinforcement learning / Imitation learning
Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes gradient explosion in deterministic-policy GAIL (DE-GAIL) versus stochastic-policy GAIL (ST-GAIL) via a probabilistic lower bound, and proposes CREDO, which clips the discriminator-based reward on expert samples to stabilize DE-GAIL training.
Problem
DE-GAIL (deterministic-policy generative adversarial imitation learning) training is frequently unstable and diverges, while ST-GAIL converges reliably but is sample-inefficient; the cause of this gradient explosion is unclear.
Approach
Model the deterministic policy as a Gaussian with vanishing covariance, derive a probabilistic lower bound on exploding policy gradients tied to expert-imitator action disparity, and mitigate it by clipping the reward (CREDO) on expert samples so that discriminator outputs stay away from 1.
Key results
- Across 11 runs per environment, DE-GAIL variants diverged in over 36% of experiments, while ST-GAIL (PPO-GAIL, TSSG) converged in all runs.
- SD3-GAIL with CREDO (clip threshold c=5) reduces outlier failures with zero-return runs compared with vanilla SD3-GAIL, per the box plots over 11 runs.
- CREDO reportedly gives data efficiency about an order of magnitude higher than PPO-GAIL, per the authors' claim.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a deterministic policy instead of a stochastic policy | GAIL · Hopper-v2, HalfCheetah-v2, Walker2d-v2 | training stability | Hurt |
| use a deterministic policy instead of a stochastic policy | GAIL · MuJoCo tasks | convergence speed: approximately ten times slower | Helped |
| use a deterministic policy instead of a stochastic policy | GAIL · MuJoCo tasks | return | Helped |
| replace Probability Logarithm Reward with the AIRL combination reward | DE-GAIL · DE-GAIL training | training stability | Helped |
| clip discriminator-outlier rewards with CREDO | SD3-GAIL · Hopper-v2, HalfCheetah-v2, Walker2d-v2 | training stability | Helped |
| clip discriminator-outlier rewards with CREDO | SD3-GAIL · Hopper-v2, HalfCheetah-v2, Walker2d-v2 | convergence | Helped |
Surprising findings
- DE-GAIL converges much faster than ST-GAIL when it does converge, yet fails frequently, a trade-off the authors attribute to gradient explosion.
- Switching the reward to the AIRL form was reported to mitigate gradient explosion in DE-GAIL.
Limitations
- Theory relies on an optimal or perturbed discriminator and a Gaussian approximation of deterministic policies with covariance tending to zero, which is an idealization.
- Experiments are limited to MuJoCo continuous-control tasks with a single expert generated by SAC.
- Clipping threshold c=5 is a fixed hyperparameter chosen without a reported sweep.
Experimental setup
Datasets: MuJoCo Hopper-v2 MuJoCo HalfCheetah-v2 MuJoCo Walker2d-v2 MuJoCo Humanoid-v2 MuJoCo Ant-v2
Benchmarks: MuJoCo Hopper-v2 MuJoCo HalfCheetah-v2 MuJoCo Walker2d-v2 MuJoCo Humanoid-v2 MuJoCo Ant-v2
Metrics: episode return divergence/failure rate absolute gradient norm P(expert) discriminator output
Baselines: DDPG-GAIL TD3-GAIL SD3-GAIL PPO-GAIL TSSG AIRL
Builds on: GAIL AIRL TSSG SD3 TD3
Closest papers by meaning
- On Computation and Generalization of Generative Adversarial Imitation Learning (ICLR 2020 · similarity 0.95)
- Generative Adversarial Imitation Learning with Neural Network Parameterization: Global Optimality and Convergence Rate (ICML 2020 · similarity 0.94)
- C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory (NeurIPS 2024 · similarity 0.94)
- Diffusion-Reward Adversarial Imitation Learning (NeurIPS 2024 · similarity 0.90)
- Diffusing States and Matching Scores: A New Framework for Imitation Learning (ICLR 2025 · similarity 0.89)
- Provably and Practically Efficient Adversarial Imitation Learning with General Function Approximation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.