Research map / Image generation / Autoregressive and latent generation

Continuous Latent Diffusion Language Model

NeurIPS 2026 · Image generation › Autoregressive and latent generation · approach: Latent diffusion with learned tokenizers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Cola DLM, a hierarchical latent diffusion language model that pairs a causal Text VAE with a block-causal DiT prior over continuous latents, and compares it to matched ~2B AR and LLaDA baselines across scaling and ablations.

Problem

Existing text generation paradigms (autoregressive, discrete diffusion, continuous token-space diffusion) struggle to jointly provide efficient non-autoregressive generation, scalable representations, and global semantic modeling.

Approach

Use diffusion not for token-level observation recovery but for transporting a continuous latent prior: a Text VAE maps text to latents, a block-causal DiT (flow matching) learns the latent prior block by block, and a conditional decoder realizes text, with the VAE co-trained with the DiT.

Key results

• Under a matched ~2B setup with a unified few-shot generative protocol, Cola DLM reaches the best Task Average at the highest compute budgets, with the paper describing steady scaling up to about 2000 EFLOPs.
• Joint VAE-DiT training from a pretrained VAE outperforms a fixed VAE and from-scratch training; learnable VAE logSNR gives Task Avg 22.1 vs 21.80 for best fixed logSNR=1.5 at 116.78 EFLOPs.
• Clean-condition repaint for the first block gives Task Avg 24.6 versus 16.7 for right padding and 8.4 for partial repaint with t=1.

Surprising findings

• Lower likelihood-derived perplexity does not necessarily yield better generated tokens; generation quality tracks latent semantic smoothness while PPL tracks probability-space smoothness.
• Block size 16 outperforms block size 1 (fully causal) in the reported setup.

Limitations

• Absolute accuracy on multiple-choice benchmarks is low under the generative few-shot protocol, making results hard to compare with standard likelihood-based evaluations.
• Comparisons use a single moderate configuration; the paper calls results conservative and leaves scaling of fixed VAE logSNR settings open.
• Likelihood estimates (PPL) are misaligned with generation quality, so the PPL-based probabilistic claims are weakly validated.

Experimental setup

Datasets: external open-source pretraining data LAMBADA MMLU SIQA SQuAD Story Cloze OBQA RACE HellaSwag

Benchmarks: LAMBADA MMLU SIQA SQuAD Story Cloze OBQA RACE HellaSwag

Metrics: accuracy (strict string matching) task average accuracy perplexity

Baselines: autoregressive LLaMA-style model LLaDA Plaid

Builds on: LLaDA LLaMA flow matching OLMo 2

Closest papers by meaning

• Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.92)
• How to Train Your Latent Diffusion Language Model Jointly With the Latent Space (NeurIPS 2026 · similarity 0.91)
• LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling (NeurIPS 2026 · similarity 0.91)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.90)
• TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings (AAAI 2025 · similarity 0.90)
• Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.89)