Research map / Reinforcement learning / RL from feedback

Block-R1: Rethinking the Role of Block Size in Multi-domain Reinforcement Learning for Diffusion Large Language Models

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL for LLM reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Block-R1, a benchmark and sample-level block-size-conditioned RL post-training method that assigns each training sample its best-improved block size to mitigate domain block size conflict in multi-domain RL for diffusion LLMs.

Problem

Multi-domain RL post-training of diffusion LLMs with a single fixed block size causes domain block size conflict, so joint training underperforms single-domain RL and can fall below the base model.

Approach

Treat block size as a sample-level training hyperparameter: pick each sample's best-improved block size via teacher-student reward gaps, store it in a 41K dataset, and condition rollouts on it during GRPO-style RL; quantify conflict with a Wasserstein-based Block Size Conflict Score.

Key results

• On LLaDA-8B-Instruct, Block-R1 raises multi-domain RL Countdown accuracy from 30.08 (Vanilla) to 62.11 and KK from 30.14 to 50.14.
• Block-R1 improves GSM8K from 57.24 to 80.74 under multi-domain RL with the same backbone.
• Across six backbones (including LLaDA2.0-16B), Block-R1 gives gains on all reported benchmarks, e.g. LLaDA2.0-16B GSM8K 70.81 to 85.52.

Surprising findings

• Vanilla multi-domain RL with a fixed block size can perform worse than the base model without RL on several benchmarks.
• Applying Block-R1 to single-domain RL also yields gains, e.g. StableDRL on KK improves by +10.00 at 256 length.

Limitations

• Block-size selection relies on teacher-student reward gaps from a specific 16B teacher and 8B base student, which may not transfer to other choices.
• Sample-level block sizes are fixed per sample and do not adapt during decoding; test-time gap to oracle remains.

Experimental setup

Datasets: GSM8K MATH500 Countdown KodCode Sudoku KK HellaSwag MMLU ARC-E MMLU-Pro ARC-C HumanEval

Benchmarks: Countdown GSM8K MATH500 HumanEval MBPP KodCode Sudoku KK HellaSwag MMLU ARC-E MMLU-Pro

Metrics: zero-shot pass@1 accuracy BCS (Wasserstein distance)

Baselines: Diffu-GRPO d1 wd1 GDPO MDPO ESPO StableDRL b1 AdaBlock-dLLM SPG

Builds on: GRPO d1 StableDRL b1

Closest papers by meaning

• Break the Block: Dynamic-size Reasoning Blocks for Diffusion Large Language Models via Monotonic Entropy Descent with Reinforcement Learning (ICML 2026 · similarity 0.91)
• d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
• Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.90)
• On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.88)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.88)
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models (NeurIPS 2025 · similarity 0.88)