Research map / Large language models / Instruction and fine-tuning

Large Language Diffusion Models

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 17 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains LLaDA, a masked diffusion language model from scratch at 8B parameters (2.3T tokens pre-training, 4.5M-pair SFT), and shows it competes with LLaMA3 8B on general benchmarks while breaking the reversal curse.

Problem

Whether the core capabilities of LLMs (scalability, in-context learning, instruction-following) depend on autoregressive modeling or can emerge from generative modeling principles via diffusion.

Approach

Replace the autoregressive formulation with a masked diffusion model: a Transformer mask predictor trained with a masked cross-entropy loss that upper-bounds the negative log-likelihood, with bidirectional conditioning and iterative low-confidence remasking at sampling time.

Key results

• LLaDA 8B Base scores 65.9 on MMLU and 70.3 on GSM8K, outperforming LLaMA3 8B Base (65.4 and 48.7) in those tables and surpassing LLaMA2 7B on nearly all tasks.
• On the Chinese poem reversal completion task, LLaDA-8B Instruct gets 45.6 reversal vs. GPT-4o's 34.3, though GPT-4o leads forward (82.7 vs 51.8).
• LLaDA 8B Instruct trails LLaMA3 8B Instruct on most metrics; e.g., GSM8K 69.4 vs 78.3 and HumanEval 49.4 vs 59.8.

Measured effects reported in the paper

Surprising findings

• Block-diffusion-style semi-autoregressive sampling gives 78.6 GSM8K and 42.2 Math for LLaDA 8B Instruct, above the pure diffusion numbers, attributed to EOS padding in SFT data causing early termination.
• LLaDA is strong on reversal reasoning, outperforming GPT-4o and Qwen2.5-7B Instruct on reversal poem completion despite weaker forward generation.

Limitations

• Fixed user-specified generation length hyperparameter; adaptive length is left for future work.
• Compute-limited ARM comparison only up to about 1e23 FLOPs, and the 8B ARM baseline is a 7B model instead.
• No RL alignment; no specialized attention or KV-cache optimization, so inference efficiency is not addressed.

Experimental setup

Datasets: web data books academic papers code math multilingual corpus SFT instruction pairs (1M human-annotated, 3.5M synthetic)

Benchmarks: MMLU BBH ARC-C Hellaswag TruthfulQA WinoGrande PIQA GSM8K Math GPQA HumanEval HumanEval-FIM

Metrics: accuracy pass@1

Baselines: LLaMA3 8B LLaMA2 7B Qwen2 7B Qwen2.5 7B Mistral 7B Deepseek 7B Gemma2 9B GPT-4o

Builds on: masked diffusion models MaskGIT BERT LLaMA3 Warmup-Stable-Decay

Closest papers by meaning

• Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.91)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.91)
• Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.91)
• UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models (ICLR 2026 · similarity 0.91)
• LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.91)
• From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs (NeurIPS 2026 · similarity 0.90)