Research map / Image generation / Autoregressive and latent generation

How to Train Your Latent Diffusion Language Model Jointly With the Latent Space

NeurIPS 2026 · Image generation › Autoregressive and latent generation · approach: Latent diffusion with learned tokenizers · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LDLM, which jointly trains a latent encoder (on frozen GPT-2 hidden states), a latent diffusion transformer, and a decoder for non-autoregressive text generation, with a four-part training recipe that makes joint training work.

Problem

Latent diffusion for text relies on latent spaces built by encoders frozen during diffusion training, so the latent space is not shaped for the diffusion objective; naive joint training of encoder and diffusion yields low-quality generation.

Approach

Train the latent encoder jointly with the diffusion model on top of frozen pre-trained GPT-2 hidden states, using an MSE hidden-state reconstruction loss, diffusion-to-encoder gradient warmup, adaptive timestep sampling, and decoder-input noise.

Key results

• On OWT-128 ablations, the full recipe reaches Gen. PPL 66.3 and Mauve 89.1, versus 98.5 for COSMOS-style and 149.7 for TEncDM.
• LDLM is reported to be 2-13x faster in sampling than competing diffusion LMs; on OWT-1024 with 1024 NFEs it samples in 40 s versus 529 s for MDLM.
• Decoder input noise has a non-monotonic effect: sigma_dec=3 is best, while sigma_dec=4 collapses Mauve to 0.4.

Surprising findings

• Naive joint training without warmup collapses the latent norm and yields Gen. PPL 403 and Mauve 0.4.
• MSE hidden-state reconstruction outperforms token-level cross-entropy reconstruction, and adding CE to MSE degrades performance.

Limitations

• Training is slower than baselines (5.1 days vs 2.9 for MDLM on OWT-1024) because the autoencoder is invoked at every step.
• Hyperparameters such as warmup length and sigma_dec are sensitive and were tuned on OWT-128, and BERT-base needed the same recipe without tuning.

Experimental setup

Datasets: OpenWebText LM1B

Benchmarks: OpenWebText LM1B

Metrics: Gen. PPL entropy diversity Mauve training time sampling time

Baselines: MDLM ReMDM Duo Psi-sampler CANDI FLM TEncDM COSMOS

Builds on: COSMOS TEncDM MDLM Perceiver Resampler Diffusion Transformer

Closest papers by meaning

• Continuous Latent Diffusion Language Model (NeurIPS 2026 · similarity 0.91)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.91)
• Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.90)
• Energy-Based Diffusion Language Models for Text Generation (ICLR 2025 · similarity 0.90)
• Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.90)
• The Diffusion Duality (ICML 2025 · similarity 0.90)