Research map / Large language models / Language, safety and interpretability
Global Context Guidance for Diffusion Large Language Models
Jeimin Jeon, Bumsub Ham
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes global context guidance for diffusion LLMs, building on weak-model guidance methods, to steer denoising predictions of the base model.
Problem
Existing guidance methods for diffusion LLMs use a weak model to guide base-model denoising, but the weak model's influence during denoising is unclear and computing its predictions is costly.
Approach
Guidance based on global context rather than a separate weak model's per-step predictions; the abstract is truncated so the exact mechanism is not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Global Context Guidance on global heads | LLaDA-8B · GSM8K-CoT | accuracy: 46.7 -> 48.4 | Improved |
| add U-CFG weak-model guidance | LLaDA-8B · GSM8K-CoT | accuracy: 46.7 -> 36.7 | Hurt |
| add A-CFG weak-model guidance | LLaDA-8B · GSM8K-CoT | accuracy: 46.7 -> 34.7 | Hurt |
| replace weak-model guidance with GCG | LLaDA-8B · inference cost | FLOPs per step: 31.8T -> 15.9T | Improved |
| guide local heads instead of global heads (LCG) (ablation) | GCG · GSM8K-CoT, Ruler | accuracy: Ruler 29.5 -> 26.0 (LLaDA-8B); GSM8K 5.4 -> 2.7 (Dream-7B) | Hurt |
| apply GCG to least sensitive blocks (LSB) (ablation) | GCG · Ruler | average score: 29.5 -> 23.4 | Hurt |
| sweep guidance scale s from 0.1 to 0.5 (ablation) | GCG on LLaDA-8B · Ruler | average score: 29.5 (s=0.3) -> 25.7 (s=0.5) | Mixed |
| swap calibration set (GSM8K-CoT, HumanEval, Ruler) for head selection (ablation) | GCG · LLaDA-8B benchmarks | accuracy | No clear effect |
| apply GCG to Llama-3.2-3B and RoBERTa-base | GCG · ARC-c, HumanEval, MBPP, SQuAD v2, GLUE | accuracy: e.g. HumanEval 51.2 -> 51.8; GLUE 84.70 -> 84.66 | Mixed |
Experimental setup
Builds on: weak-model guidance for diffusion LLMs
Closest papers by meaning
- UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models (ICLR 2026 · similarity 0.92)
- Uncertainty Quantification for Large Language Diffusion Models (NeurIPS 2026 · similarity 0.91)
- Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (ICLR 2025 · similarity 0.90)
- Denoising Time Matters: Diverse Generation in Diffusion Language Models (NeurIPS 2026 · similarity 0.90)
- From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs (NeurIPS 2026 · similarity 0.90)
- Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
