Research map / Reinforcement learning / Policy gradient and actor-critic

Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matching

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AMDP, which trains diffusion policies for maximum-entropy RL via reciprocal adjoint matching with simulation-free regression on stored terminal actions, plus erf squashing and a trust-region loss.

Problem

Training diffusion policies for online maximum-entropy RL is hard without ground-truth action data; reverse-KL methods require backpropagation through the full diffusion chain, and Q-score/importance-weighted methods lack guarantees or scale poorly.

Approach

Frames diffusion policy optimization for MaxEnt-RL as stochastic optimal control and uses reciprocal adjoint matching, a fixed-point regression on the Q-score of terminal actions that can be trained simulation-free from a replay buffer; adds erf squashing whose Jacobian cancels the reference density and a Lagrangian trust-region update.</îcore_idea>

Key results

• On MuJoCo Playground humanoid tasks AMDP learns significantly faster than baselines; with best-of-16 sampling (AMDP BoN) it surpasses all baselines, though REPPO eventually slightly surpasses AMDP without BoN.
• On HumanoidBench AMDP outperforms all baselines by a large margin even without best-of-N sampling.
• Wallclock overhead is roughly 10% over REPPO for AMDP, while reverse-KL optimization is about 10x slower at 16 diffusion steps and 72-80x slower at 128 steps.

Surprising findings

• Reverse-KL (DIME-style) optimization performs poorly on MuJoCo Playground humanoid tasks, indicating need for trust-region updates.

Limitations

• Requires deterministic initial condition X0=0 for memorylessness, which complicates offline-to-online pretraining with Gaussian priors.
• Relies on trust-region tuning and on a Q-function estimate for the score target.

Experimental setup

Datasets: MuJoCo Playground DMC MuJoCo Playground humanoid ManiSkill3 HumanoidBench DMC dog and humanoid

Benchmarks: MuJoCo Playground DMC MuJoCo Playground humanoid ManiSkill3 HumanoidBench DMC dog and humanoid

Metrics: IQM return wallclock time per iteration

Baselines: REPPO PPO SPO DPPO FPO DIME QSM Diff-QL Consistency-AC

Builds on: adjoint matching reciprocal adjoint matching DIME REPPO QSM

Closest papers by meaning

• DIME: Diffusion-Based Maximum Entropy Reinforcement Learning (ICML 2025 · similarity 0.93)
• Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.91)
• Maximum Entropy Reinforcement Learning with Diffusion Policy (ICML 2025 · similarity 0.91)
• Diffusion Actor-Critic with Entropy Regulator (NeurIPS 2024 · similarity 0.90)
• Maximum Entropy Inverse Reinforcement Learning of Diffusion Models with Energy-Based Models (NeurIPS 2024 · similarity 0.90)
• Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (NeurIPS 2026 · similarity 0.90)