Research map / Reinforcement learning / RL from feedback

Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods

NeurIPS 2025 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Score Entropy Policy Optimization (SEPO), a policy-gradient method for fine-tuning discrete diffusion models via concrete scores, using SNIS estimation of marginals and a clipped-ratio loss, with a gradient-flow variant.

Problem

Fine-tuning discrete diffusion models with non-differentiable rewards (as in RLHF) is hard because categorical sampling is non-differentiable and the policy gradient requires marginals that are intractable to estimate.

Approach

Derives a discrete REINFORCE/TRPO-style gradient in the concrete score framework, estimates the marginal q(y) via self-normalized importance sampling (harmonic mean of conditionals), and applies PPO/GRPO-style clipping and advantages; a gradient-flow sampling variant uses a time-homogeneous CTMC with corrector steps.

Key results

• On DNA enhancer design, SEPO with gradient flow reaches Pred-Activity 7.64 and ATAC-Acc 99.9%, the best among compared baselines including DRAKES (5.61, 92.5%) and GLIDE2E (7.35, 90.6%).
• On language modeling, SEDD-SEPO-1024 is preferred over the vanilla SEDD by a GPT-3.5 judge; SEDD-SEPO-1024 wins are reported at up to 68.8% for the 512-step SEDD-SEPO-128 variant vs vanilla.
• SEPO runs with fewer epochs (14 for SEPO, 8 with GF) than DRAKES (66-127), with runtime around 19-24 minutes on one RTX 3090.

Measured effects reported in the paper

Surprising findings

• Without SNIS (naive single-sample estimate of q(y)), the method reaches only 4.66 Pred-Activity despite running 199 epochs, showing the importance of the SNIS estimator.
• SEPO with gradient flow achieves higher Pred-Activity and ATAC-Acc but lower log-likelihood (-248.2) and 3-mer correlation is lower than DRAKES.

Limitations

• Inner importance sampling requires resampling for each neighbor of outer samples, adding computational cost.
• Theoretical assumptions (bounded gradients, lower-bounded probabilities, KL-Lipschitz Gamma) are strong and the convergence rate is O(1/sqrt(S)) only on averaged gradient norms.
• Language experiments are small-scale and evaluated only with a single LLM judge over 153 prompts.

Experimental setup

Datasets: HH-RLHF Awesome ChatGPT Prompts Gosai dataset

Benchmarks: Awesome ChatGPT Prompts pairwise comparison DNA enhancer activity (Pred-Activity) ATAC-Acc 3-mer Corr Log-Lik

Metrics: win rate by judge median predicted activity ATAC accuracy 3-mer correlation log-likelihood

Baselines: DRAKES GLIDE2E CG SMC TDS CFG

Builds on: PPO GRPO TRPO REINFORCE DRAKES

Closest papers by meaning

• Score as Action: Fine Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning (ICML 2025 · similarity 0.90)
• Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (ICML 2024 · similarity 0.89)
• Score Regularized Policy Optimization through Diffusion Behavior (ICLR 2024 · similarity 0.88)
• Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages (NeurIPS 2026 · similarity 0.88)
• Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence (ICML 2025 · similarity 0.88)
• Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model (CVPR 2024 · similarity 0.88)