Research map / Reinforcement learning / Policy gradient and actor-critic

Does This Gradient Spark Joy?

NeurIPS 2026 · Reinforcement learning › Policy gradient and actor-critic · approach: Trajectory reuse in policy gradients · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the Kondo gate, which uses forward-pass delight (advantage × surprisal) with a compute price to stochastically decide which samples get a backward pass, cutting backward passes by ~30x-100x while keeping most of DG's learning quality on MNIST and token reversal.

Problem

Policy gradient spends an expensive backward pass on every sample, though most samples carry little learning value; the paper asks whether a forward-pass signal can decide which backward passes to skip.

Approach

Gate each sample's backward pass with a Bernoulli draw whose probability is sigmoid((delight − price)/temperature), so delight decides whether a gradient is computed at all rather than only how much it is weighted; the price is set adaptively to hit a target gate rate.

Key results

• On MNIST at gate rate 0.03, the Kondo gate nearly matches full DG in forward-pass space while using only 3% of backward passes, dominating by two orders of magnitude in backward-pass space (30 seeds).
• At a 4x backward/forward cost ratio on MNIST, DG-K reaches 5% test error about 6x faster than PG in compute.
• On token reversal, DG-K (rho=3%) solves H* ≈ 29 in backward-pass space, while DG needs compute for H* ≈ 27 at far higher backward cost.

Surprising findings

• Under a zero baseline on MNIST, DG-K outperforms full DG in forward-pass space.
• Fixed 3% gate rate becomes too aggressive at large vocabulary sizes, while the adaptive lambda=0 gate tracks DG.

Limitations

• Gambling regime (high reward variance on rare suboptimal actions) makes delight produce false breakthroughs; DG collapses sharply near sigma_G ≈ 1 on MNIST.
• Experiments are limited to MNIST bandits and small token reversal transformers; scaling to modern large-model training is untested.
• DG-K is more fragile than DG to noise in delight and forward-pass logits.

Experimental setup

Datasets: MNIST token reversal (synthetic)

Benchmarks: MNIST contextual bandit test error token reversal learning curves and solved length/vocabulary

Metrics: test classification error reward / error vs forward passes reward vs backward passes largest solved sequence length H* largest solved vocabulary M* compute speedup vs PG

Baselines: PG PPO PMPO DG advantage-only priority surprisal-only priority additive priority mix uniform subsampling

Builds on: Delightful Policy Gradient REINFORCE PPO PMPO GRPO

Closest papers by meaning

• Delightful Distributed Policy Gradient (NeurIPS 2026 · similarity 0.87)
• FLOPS: Forward Learning with OPtimal Sampling (ICLR 2025 · similarity 0.85)
• Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems (NeurIPS 2025 · similarity 0.84)
• Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates (NeurIPS 2024 · similarity 0.84)
• Feel-Good Thompson Sampling for Contextual Bandits: a Markov Chain Monte Carlo Showdown (NeurIPS 2025 · similarity 0.83)
• Directly Fine-Tuning Diffusion Models on Differentiable Rewards (ICLR 2024 · similarity 0.83)