Research map / Large language models / Language, safety and interpretability

From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Parallel and early-exit decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes adapting off-the-shelf AR LLMs into block diffusion LMs via context-causal attention, a one-pass parallel training mask with auxiliary AR loss, and gradual block-size growth, yielding NBDiff-7B.

Problem

Adapting pretrained autoregressive LLMs into diffusion language models is hard: prior methods either randomly anneal attention masks toward full-sequence diffusion or naively transplant AR weights into block diffusion, leaving the adaptation destination and path unclear.

Approach

Treat AR as block diffusion with block size 1 and make block diffusion the destination; keep the context strictly causal while the active block is bidirectional, train all blocks in one forward pass with a structured mask, add an auxiliary AR loss on the clean context, and grow the block size gradually from 1 to 32.

Key results

• NBDiff-7B-Instruct reaches a 79.9 average across general, math and coding benchmarks, above SDAR-8B (74.0) and LLaDA2.0-mini (71.0), with 84.0 on MATH and 89.0 on HumanEval.
• NBDiff-7B-Base reaches a 65.3 average, above Dream-v0-Base-7B (60.1) and LLaDA-8B (52.0), with 79.6 GSM8K and 46.0 MATH.
• On Qwen3-4B, the full method improves average accuracy from 47.56 (annealed attention mask) to 50.28; on openPangu-7B, adding AR loss and gradual growth raises plain finetuning from 48.95 to 54.94.

Surprising findings

• Adapting from the Base AR checkpoint gave a slightly better overall balance than adapting from the SFT checkpoint (Avg 53.26 vs 54.39 on Qwen3-8B, with the SFT edge driven largely by HumanEval).
• Block-causal attention in the context performed far worse than context-causal in the preliminary adaptation (MATH 1.6 vs 36.8).

Limitations

• Main 7B results rely on an internal 700B-token dataset and internal SFT data, limiting reproducibility.
• Gains on individual benchmarks are mixed (e.g., HumanEval for Qwen3-4B and IFEval for NBDiff-7B-Instruct trail some baselines).
• Hardware and compute details, and the exact block-growth schedule hyperparameters, are not fully specified.

Experimental setup

Datasets: FineWeb-100B openPangu-Embedded-7B training dataset (700B internal) 10B-token SFT data

Benchmarks: MMLU MMLU-Pro CMMLU CEval IFEval GSM8K MATH MBPP HumanEval BBH

Metrics: accuracy average accuracy

Baselines: Annealed Attention Mask Plain Finetuning LLaDA-8B LLaDA-MoE-7B LLaDA2.0-mini Dream-v0-Instruct-7B Dream-v0-Base-7B SDAR-8B

Builds on: LLaDA Block-Diffusion DiffuLLaMA Fast-DLLM-v2 SDAR

Closest papers by meaning

• Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.93)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.92)
• Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.92)
• SemanticDLM+: Improving Diffusion LLMs through Bias-variance Trade-off in Transition Kernel Design (NeurIPS 2026 · similarity 0.92)
• Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.92)
• UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models (ICLR 2026 · similarity 0.91)