Research map / Reinforcement learning / Policy gradient and actor-critic
GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenPO, an on-policy PPO-based method that trains flow/diffusion policies by making the flow exactly invertible via a doubled dummy-action EDICT-style inversion, enabling exact log-likelihoods, entropy and KL estimates on IsaacLab tasks.
Problem
Diffusion and flow policies lack tractable state-action log-likelihoods because forward and reverse sampling are not exactly invertible under Euler-Maruyama discretization, which blocks their use in on-policy RL such as PPO on massively parallel simulators.
Approach
Construct an exactly invertible flow using EDICT-style coupled dummy-action pairs (x, y) with alternating updates and a mixing step, so change-of-variables gives exact log-likelihood; the executed action is the average of the pair. This enables unbiased entropy and KL estimates for KL-adaptive learning rates and entropy regularization in PPO, plus a compression loss on x-y discrepancy.
Key results
- GenPO achieved the highest final average return on all eight IsaacLab tasks, e.g., 177.90 on Ant vs 146.94 for PPO and 273.94 on Humanoid vs 197.25 for PPO.
- On Shadow-Hand GenPO reached 11282.35 versus 9386.64 for TD3 and 8402.21 for PPO.
- Off-policy diffusion baselines DACER and QVPO performed poorly in the massively parallel setting, often below PPO and sometimes negative on Ant and Franka-Arm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GenPO instead of PPO | on-policy RL · Ant | average final episodic return: 146.94(10.61) -> 177.90(13.87) | Helped |
| Use GenPO instead of PPO | on-policy RL · Humanoid | average final episodic return: 197.25(18.26) -> 273.94(16.96) | Helped |
| Use GenPO instead of PPO | on-policy RL · Franka Arm | average final episodic return: 78.14(50.43) -> 144.78(3.10) | Helped |
| Use GenPO instead of PPO | on-policy RL · Quadcopter | average final episodic return: 99.08(13.49) -> 137.95(0.84) | Helped |
| Use GenPO instead of PPO | on-policy RL · Anymal-D | average final episodic return: 9.80(4.78) -> 19.80(0.16) | Helped |
| Use GenPO instead of PPO | on-policy RL · Unitree-Go2 | average final episodic return: 15.67(1.92) -> 28.01(0.76) | Helped |
| Use GenPO instead of PPO | on-policy RL · Unitree-H1 | average final episodic return: 18.97(2.02) -> 26.09(0.68) | Helped |
| Use GenPO instead of TD3 | on-policy RL · Shadow Hand | average final episodic return: 9386.64(314.37) -> 11282.35(322.94) | Helped |
| Add compression loss with coefficient ν=0.01 (ablation) | GenPO · Ant | convergence speed and final performance | Helped |
| Increase compression-loss coefficient beyond ν=0.01 (ablation) | GenPO · Ant | final performance | Hurt |
| Add adaptive learning-rate adjustment (ablation) | GenPO · Ant | convergence speed and stability | Helped |
| Add entropy regularization (ablation) | GenPO · Ant | episodic return | Helped |
Surprising findings
- Off-policy diffusion-based RL methods (DACER, QVPO) are nearly ineffective in massively parallel IsaacLab settings, with several returns near zero or negative.
- Larger compression-loss coefficients over-constrain the policy and degrade performance, while no compression loss slows convergence.
Limitations
- Relatively high computational and memory overhead due to Jacobian determinant computation, as stated by the authors.
- Doubled dummy action space introduces redundant exploration that requires a compression loss and mixing coefficient tuning.
- Evaluated only in simulation (IsaacLab); no real-robot deployment reported.
Experimental setup
Datasets: IsaacLab Isaac-Ant-v0 Isaac-Humanoid-v0 Isaac-Lift-Cube-Franka-v0 Isaac-Repose-Cube-Shadow-Direct-v0 Isaac-Velocity-Flat-Anymal-D-v0 Isaac-Velocity-Rough-Unitree-Go2-v0 Isaac-Velocity-Rough-H1-v0 Isaac-Quadcopter-Direct-v0
Benchmarks: IsaacLab Ant IsaacLab Humanoid IsaacLab Franka-Arm IsaacLab Quadcopter IsaacLab Anymal-D IsaacLab Unitree-Go2 IsaacLab Unitree-H1 IsaacLab Shadow-Hand
Metrics: average episodic return standard deviation over 5 seeds
Baselines: DDPG TD3 SAC PPO DACER QVPO
Builds on: PPO EDICT DPPO DACER QVPO
Closest papers by meaning
- Diffusion Policy Policy Optimization (ICLR 2025 · similarity 0.91)
- Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.90)
- TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment (NeurIPS 2026 · similarity 0.89)
- One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation (ICML 2025 · similarity 0.89)
- OMPO: A Unified Framework for RL under Policy and Dynamics Shifts (ICML 2024 · similarity 0.89)
- Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.