Research map / Image generation / Autoregressive and latent generation
Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes A-CFG, which builds the unconditional branch of classifier-free guidance by re-masking the lowest-confidence generated tokens at each step of a masked diffusion language model, and reports gains over standard CFG on LLaDA 8B and Dream-7B.
Problem
Standard classifier-free guidance in iterative masked diffusion language models uses a static unconditional input (null prompt or fully masked target), which may not match the model's changing uncertainty across tokens and denoising steps.
Approach
At each denoising step, tokens already generated with low max-softmax confidence are temporarily re-masked to form a dynamic, localized unconditional input, and the standard CFG formula is applied on top of the conditional logits.
Key results
- On LLaDA 8B, A-CFG reaches 33.3 on GPQA versus 29.4 for standard CFG (+3.9) and 42.0 on Sudoku versus 34.0 (+8.0).
- A-CFG improves LLaDA 8B over standard CFG on most listed benchmarks, e.g., GSM8K 73.5 vs 70.8.
- Applied to Dream-7B, A-CFG raises Sudoku from 72.0 to 80.0 and ARC-C from 59.8 to 60.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace static unconditioning with adaptive low-confidence masking | LLaDA 8B · GPQA | accuracy: 29.4 -> 33.3; +3.9 points | Helped |
| Replace static unconditioning with adaptive low-confidence masking | LLaDA 8B · Sudoku | accuracy: 34.0 -> 42.0; +8.0 points | Helped |
| Replace static unconditioning with adaptive low-confidence masking | LLaDA 8B · GSM8K | accuracy: 70.8 -> 73.5; +2.7 points | Helped |
| Replace static unconditioning with adaptive low-confidence masking | Dream-7B · Sudoku | accuracy: 72.0 -> 80.0; +8.0 points | Helped |
| Replace static unconditioning with adaptive low-confidence masking | Dream-7B · ARC-C | accuracy: 59.8 -> 60.8; +1.0 point | Helped |
| Increase re-masking proportion from 0.1 to 0.7 (ablation) | LLaDA 8B (A-CFG) · ARC-C | accuracy: 45.9 -> 47.8 | Helped |
| Increase re-masking proportion from 0.7 to 0.9 (ablation) | LLaDA 8B (A-CFG) · ARC-C | accuracy: 47.8 -> 46.0 | Hurt |
| Increase guidance scale from 0.0 to 0.5 (ablation) | LLaDA 8B (A-CFG) · ARC-C | accuracy: 45.5 -> 47.8 | Helped |
| Increase guidance scale from 1.0 to 1.5 or 2.0 (ablation) | LLaDA 8B (A-CFG) · ARC-C | accuracy: 47.8; 47.5; 47.6 | Hurt |
Surprising findings
- On ARC-C, the guidance scale w=0 (re-masking only, no CFG extrapolation) already yields 45.5, above the no-guidance LLaDA baseline of 45.5 tied, and performance is fairly flat across w from 0.5 to 2.0.
Limitations
- Gains over standard CFG are modest on several general benchmarks (e.g., +0.3 on MMLU), and guidance scale and re-masking proportion were tuned on benchmark validation sets.
- Compute overhead of the extra unconditional forward pass per step is not quantified.
- Evaluated only on two masked diffusion models; the confidence measure and rho heuristic are not compared against alternatives beyond a brief note.
Experimental setup
Benchmarks: MMLU BBH ARC-C Hellaswag TruthfulQA WinoGrande PIQA GSM8K MATH GPQA Countdown Sudoku
Metrics: accuracy exact match
Baselines: Std CFG No Guidance LLaMA3 8B LLaMA2 7B Qwen2 7B
Builds on: classifier-free guidance LLaDA low-confidence remasking
Closest papers by meaning
- Stay on Topic with Classifier-Free Guidance (ICML 2024 · similarity 0.91)
- Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models (AAAI 2025 · similarity 0.90)
- Visual Generation Without Guidance (ICML 2025 · similarity 0.88)
- Diffusion Models without Classifier-free Guidance (NeurIPS 2026 · similarity 0.88)
- Feedback Guidance of Diffusion Models (NeurIPS 2025 · similarity 0.88)
- Towards a Golden Classifier-Free Guidance Path via Foresight Fixed Point Iterations (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.