Research map / Large language models / Reasoning and chain of thought

DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DACA-GRPO, which adds denoising-progress-based per-token advantage weighting (DPS) and a stratified masking likelihood estimator (SML) to GRPO-style RL for diffusion LLMs, improving Diffu-GRPO, wd1, and GDPO across math, code, constraint, and JSON tasks.

Problem

Existing RL methods for diffusion LLMs assign uniform credit across denoising steps and rely on mean-field (fully masked, zero-context) likelihood estimates that are biased and high-variance for policy optimization.

Approach

Reuse logits at still-masked positions across denoising steps to compute per-step belief-progress deltas (DPS) that modulate per-token advantages at no extra forward-pass cost, and estimate sequence likelihood by stratifying output positions into K strata so each token is predicted with (K-1)/K context (SML).

Key results

• DACA-GRPO on wd1 improves MBPP at L=512 to 50.7 and reaches 78.47% JSON schema adherence (+5.90pp).
• wd1 + DPS raises Sudoku per-cell accuracy at L=512 from 34.6% to 70.9%.
• Gains reported up to 5.6pp on math, 7.4pp on code, 36.3pp on constraint satisfaction, and 5.9pp on JSON.

Surprising findings

• SML is harmful on short tightly constrained outputs (Sudoku, Countdown) because partial-mask context drives trivial gap-filling, so it is excluded there.
• Only the Extrapolate last-step mode and per-step normalization exceed the baseline across all stride settings; Raw and Neutral fail at length 512.

Limitations

• SML is not applicable to short, tightly constrained outputs due to distribution mismatch with fully-masked generation.
• SML requires K extra forward passes per loss computation.
• JSON dataset is synthetic and not yet released.

Experimental setup

Datasets: MATH-500 GSM8K KodCode Countdown Sudoku JSONSchemaBench

Benchmarks: MATH-500 GSM8K MBPP HumanEval Countdown Sudoku JSONSchemaBench github-medium

Metrics: pass@1 accuracy per-cell accuracy schema adherence

Baselines: LLaDA-8B-Instruct Diffu-GRPO wd1 GDPO

Builds on: GRPO Diffu-GRPO wd1 GDPO LLaDA

Closest papers by meaning

• Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages (NeurIPS 2026 · similarity 0.91)
• Training Diffusion Models with Reinforcement Learning (ICLR 2024 · similarity 0.89)
• Diffusion-State Policy Optimization for Masked Diffusion Language Models (NeurIPS 2026 · similarity 0.89)
• Sketch Then Paint: Hierarchical Reinforcement Learning for Diffusion Multi-Modal Large Language Models (NeurIPS 2026 · similarity 0.88)
• MRO: Enhancing Reasoning in Diffusion Language Models via Multi-Reward Optimization (NeurIPS 2025 · similarity 0.88)
• Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models (NeurIPS 2025 · similarity 0.88)