Research map / Reinforcement learning / Policy gradient and actor-critic

Delightful Distributed Policy Gradient

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Variance-reduced policy gradients · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a sign-aware surprisal gating of policy-gradient updates (Delightful Policy Gradient) that suppresses high-surprisal failures and preserves rare successes without behavior probabilities, validated on MNIST with staleness, tabular bandits, and transformer token reversal with distributed frictions.

Problem

Distributed RL trains on stale, buggy, or mismatched actor data that yields high-surprisal actions; negative updates from surprising failures dominate finite-batch updates, and importance-weighting corrections need behavior probabilities that are often unavailable or unreliable.

Approach

Gate each policy-gradient term by a sigmoid of delight (advantage times learner-policy surprisal), so rare failures get suppressed and rare successes preserved; uses only the learner's current policy, no behavior probabilities. The paper proves that sign-blind learner-probability-only gates cannot do this separation.

Key results

• On MNIST at delay D=1000, DG reaches about 2% error after 10k steps versus about 8% for importance-weighted PG, without importance weights.
• Under combined friction after 10k episodes, DG solves sequences up to H*≈13 versus ≈8 for PMPO, ≈5 for PPO, and ≈3.5 for PG.
• Under actor bugs at p_E=3e-3, DG converges to low error while baselines plateau above 10%; the paper reports DG often achieves nearly order-of-magnitude lower error.

Surprising findings

• Under a zero baseline on MNIST, uncorrected REINFORCE matches DG while importance-weighted PG becomes worse than uncorrected REINFORCE.
• Importance weighting can downweight off-policy successes exactly when the learner assigns them low probability, discarding high-delight samples.

Limitations

• Experiments are small-scale diagnostics, not large asynchronous actor-learner evaluation.
• Main failure mode is false delight: high-surprisal positive-advantage samples from reward variance, verifier error, or reward hacking can be wrongly amplified.
• Tabular suppression result is a perpendicular second-moment argument rather than a population cosine collapse; negative updates can still help in single softmax bandits.

Experimental setup

Datasets: MNIST synthetic token reversal

Benchmarks: MNIST token reversal

Metrics: classification error sequence error cosine misalignment largest solved sequence length H*

Baselines: REINFORCE importance-weighted PG PPO PMPO

Builds on: Delightful Policy Gradient PPO PMPO GRPO V-trace

Closest papers by meaning

• Adaptive Robust Estimator for Policy Optimization in Reinforcement Learning (NeurIPS 2026 · similarity 0.87)
• Does This Gradient Spark Joy? (NeurIPS 2026 · similarity 0.87)
• Beyond the Dirac Delta: Mitigating Diversity Collapse in Reinforcement Fine-Tuning for Image Generation (NeurIPS 2026 · similarity 0.87)
• Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
• ContraDiff: Planning Towards High Return States via Contrastive Learning (ICLR 2025 · similarity 0.87)
• Blending Imitation and Reinforcement Learning for Robust Policy Improvement (ICLR 2024 · similarity 0.87)