Research map / Reinforcement learning / Policy gradient and actor-critic

Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularization

NeurIPS 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Policy gradients for special settings · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DrAC, an actor-critic method that trains intractable stochastic-mapping multimodal actors (amortized and diffusion) via reparameterized policy gradient plus a log-L2-distance diversity regularizer, avoiding decision probabilities.

Problem

Continuous RL typically uses deterministic or unimodal Gaussian actors that cannot express multimodal decision distributions, and existing online multimodal methods use intractable actors whose probabilities (needed for entropy) are unavailable, hurting performance, diversity and efficiency.

Approach

Unifies amortized and diffusion actors as stochastic-mapping actors f(s,z) with fixed latent noise, optimizes them by reparameterized policy gradient, and replaces entropy with a geometric-mean-style log pairwise L2 distance diversity regularizer whose coefficient is auto-tuned to a target diversity.

Key results

• In multi-goal PointMaze, DrAmort shows the best few-shot robustness against goal removal and obstacles across all mazes, with SQL the best among remaining baselines.
• In MuJoCo, DrAmort achieves best performance on three of six tasks and surpasses SQL on five.
• In game level generation, DrAmort attains the best episodic returns for both MarioPuzzle and MultiFacet styles; DrAmort trains about as fast as SAC (21.0s vs 22.7s per 1000 steps).

Measured effects reported in the paper

Surprising findings

• Diffusion actors (DrDiffus, DACER) do not outperform the unimodal SAC on MuJoCo or game generation despite greater complexity.
• SQL, which uses SVGD gradient estimation, performs poorly on many benchmarks including MuJoCo, suggesting reparameterization is a better gradient estimator.

Limitations

• Only a basic implementation of DrAC is investigated; temperature scheduling and better stochastic-mapping actor designs are left open.
• Diffusion actors underperform, possibly due to insufficient tuning or network capacity; DrDiffus is sensitive to temperature on Humanoid.
• Diversity is measured with a simple distance metric; quantitative in-depth diversity analysis of game levels is left for future work.

Experimental setup

Datasets: D4RL PointMaze (custom multi-goal variant) MuJoCo-v4 locomotion tasks game level generation benchmark with pretrained GAN decoder

Benchmarks: multi-goal PointMaze (simple, medium, hard) MuJoCo-v4 six locomotion tasks game content generation (MarioPuzzle, MultiFacet)

Metrics: success rate number of reachable goals five-episode success rate under removal and obstacle perturb episodic return average pairwise Hamming distance hypervolume training time per 1000 steps

Baselines: SAC SQL DACER S$^2$AC

Builds on: soft actor-critic soft Q-learning DACER reparameterized policy gradient maximum-entropy RL

Closest papers by meaning

• Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradient (NeurIPS 2024 · similarity 0.90)
• Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization (NeurIPS 2024 · similarity 0.86)
• MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning (CVPR 2026 · similarity 0.86)
• Beyond SFT-to-RL: Pre-alignment via Black-box On-policy Distillation for Multimodal RL (NeurIPS 2026 · similarity 0.86)
• Perception-Aware Policy Optimization for Multimodal Reasoning (ICLR 2026 · similarity 0.85)
• Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners (NeurIPS 2025 · similarity 0.85)