Research map / Optimization / Optimizers
Adaptive-Gradient Policy Optimization: Enhancing Policy Learning in Non-Smooth Differentiable Simulations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AGPO, which adaptively mixes simulation gradients with Q-function gradients via inverse-variance weighting to make policy learning robust to non-smooth differentiable simulation dynamics.
Problem
Policy learning through differentiable simulators degrades when simulated dynamics are non-smooth or discontinuous (e.g., hard contacts), as first-order simulation gradients become high-variance and outlier-prone.
Approach
Blend first-order simulation gradients (SG) with Q-function gradients (QG) using an inverse-variance adaptive mixture ratio computed per step, so unreliable SG transitions are down-weighted in favor of the low-variance QG; truncate trajectories when cumulative credit drops below a threshold.
Key results
- AGPO outperforms SHAC, BPTT, and AoBG on five Brax control tasks and competes with or beats PPO, SAC, and TD3 in episode reward over 4M timesteps.
- Q-function gradients show variance stable across several orders of magnitude lower than simulation gradient outliers in the Brax Ant case study.
- On alternative backends (Brax positional/spring, DFlex), AGPO consistently surpasses or matches SHAC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive mixing of simulation gradients (SG) and Q gradients (QG) | AGPO · Brax generalized: Hopper, Half-Cheetah, Walker2D, Ant, Humanoid | episode reward | Helped |
| use QG instead of SG alone (α ≡ 0) (ablation) | AGPO · Brax generalized control tasks | episode reward | Helped |
| use SG alone (α ≡ 1) instead of adaptive SG-QG mixing (ablation) | AGPO · Brax generalized control tasks | episode reward | Hurt |
| replace adaptive mixing with a fixed mixture ratio (ablation) | AGPO · Ant and Humanoid | episode reward | Hurt |
| increase horizon length from h=8 toward h=128 (ablation) | AGPO · Ant | episode reward | Helped |
| regulate SG outliers with MaxNorm clipping or quantile filtering instead of adaptive SG-QG mixing (ablation) | AGPO · Ant and Humanoid | episode reward | Hurt |
| use adaptive SG-QG mixing rather than SHAC | AGPO · Brax positional and spring backends; DFlex Humanoid and HumanoidMTU | episode reward | Helped |
Surprising findings
- Policy gradient estimates showed larger fluctuating empirical variance than simulation gradients in some rollouts, contrary to prior PIPPS-style assumptions.
- Fixed-ratio SG-only (alpha=1) performed poorly, and simple gradient clipping or quantile filtering did not fix non-smooth policy learning.
Limitations
- Requires additional backpropagation steps to compute the mixture ratio.
- Uses scalar inverse-variance weighting; richer gradient composition (covariance, tensor weights) left to future work.
Experimental setup
Datasets: Brax Hopper Brax Half-Cheetah Brax Walker2D Brax Ant Brax Humanoid DFlex Humanoid DFlex HumanoidMTU
Benchmarks: Brax Hopper Brax Half-Cheetah Brax Walker2D Brax Ant Brax Humanoid DFlex Humanoid DFlex HumanoidMTU
Metrics: episode rewards empirical gradient variance wall-clock time
Baselines: BPTT SHAC AoBG PPO SAC TD3
Builds on: SHAC PIPPS Alpha-order batched gradient SAC TD3
Closest papers by meaning
- PODS: Policy Optimization via Differentiable Simulation (ICML 2021 · similarity 0.88)
- A Parametric Class of Approximate Gradient Updates for Policy Optimization (ICML 2022 · similarity 0.88)
- Gradient Informed Proximal Policy Optimization (NeurIPS 2023 · similarity 0.88)
- GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.86)
- Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.