Research map / Image generation / Autoregressive and latent generation

Next Semantic Scale Prediction via Hierarchical Diffusion Language Models

NeurIPS 2025 · Image generation › Autoregressive and latent generation · approach: Latent diffusion with learned tokenizers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HDLM, a discrete diffusion language model whose forward process moves tokens through a hierarchical cluster vocabulary before masking, enabling time-varying next-semantic-scale prediction with a closed-form ELBO that reduces to MDLM when there is one cluster.

Problem

Masked discrete diffusion gives all masked tokens identical embeddings and cannot self-correct decoded tokens, while uniform diffusion underperforms; a discrete diffusion LM with richer intermediate noise states and self-correction is needed.

Approach

Insert intermediate cluster hierarchies between clean words and the mask token (x -> c -> m), where each token is surjectively mapped to a coarser cluster token, and train with a closed-form CTMC ELBO that decomposes into cluster-level and token-level cross-entropy terms; optional stochastic cluster corruption and forced within-cluster transitions improve robustness.

Key results

• HDLM-small-128 reaches validation PPL <=23.25 vs. 27.39 for reimplemented MDLM-small at 131B tokens on OWT.
• HDLM-small-64 achieves the best generative PPL of 144.2 vs. 163.7 for MDLM-small (reimpl.).
• With stochastic cluster perturbation xi=0.8, HDLM-small-64 reduces generative PPL to 54.15 (about 62% lower than 144.2).

Measured effects reported in the paper

Surprising findings

• Setting one cluster recovers MDLM performance, and the validation PPL vs. number of clusters is non-monotonic, first rising then falling.
• Hard token-level cross-entropy training for cluster-stage denoising slightly hurts performance, suggesting progressive denoising is beneficial.

Limitations

• Experiments only at small (170M) and base (425M) scale on OpenWebText.
• Multi-level hierarchies (more than one intermediate level) are only theoretically developed, not experimentally validated.
• Generative perplexity is reported with gpt2-large as reference, and lower gen PPL can come with lower diversity under stochastic perturbation.

Experimental setup

Datasets: OpenWebText

Benchmarks: OpenWebText validation set

Metrics: validation perplexity generative perplexity

Baselines: MDLM GIDD+ SEDD GPT-2 Llama 110M

Builds on: MDLM GIDD D3PM DiT

Closest papers by meaning

• Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.91)
• SemanticDLM+: Improving Diffusion LLMs through Bias-variance Trade-off in Transition Kernel Design (NeurIPS 2026 · similarity 0.90)
• Energy-Based Diffusion Language Models for Text Generation (ICLR 2025 · similarity 0.89)
• Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.89)
• Theoretical Benefit and Limitation of Diffusion Language Model (NeurIPS 2025 · similarity 0.89)
• Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.89)