Research map / Probabilistic and scientific modeling / Variational and generative sampling

TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Models

NeurIPS 2026 · Probabilistic and scientific modeling › Variational and generative sampling · approach: Annealing and divergence-based variational objectives · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes TUBE, a tangent-linearized variational upper bound on log-likelihood with an unbiased Monte Carlo estimator, to localize the likelihood of block masked diffusion models and block any-order ARMs together with ELBO, and finds they lie strictly above the exact ARM perplexity.

Problem

Log-likelihood of discrete diffusion models and any-order autoregressive models is intractable; ELBO-based evaluation does not show how far the true value lies above, and existing upper-bound estimators (CUBO, TVO, IS-VG-B) are biased at finite samples.

Approach

Linearize log via the tangent bound log a <= log b + (a-b)/b with a surrogate psi(x); since the bound is affine in p(x)=E_pi[p(x|pi)], the Monte Carlo average gives an unbiased, bound-preserving upper-bound estimator, with surrogates either an exact ARM or a self-surrogate averaging model likelihoods over several orderings.

Key results

• On OWT with L'=4 (exact ELBO_K), block AO-ARM reaches 18.46 PPL vs exact ARM 17.54; TUBE gives >=17.74 for AO-ARM, a gap of 0.20.
• For block MDM on OWT (L'=4), TUBE gap to ARM shrinks from 80.24 PPL at NFE=1 to 1.62 at NFE=4 and TUBE is >=19.16 at NFE=4.
• TVO_U and IS-VG-B exceed ELBO_K in all NFE>=2 settings, showing they fail to give reliable bounds in practice, while TUBE stays below ELBO_K.

Surprising findings

• CUBO_beta=2 sits below ELBO_K in every row despite being labeled an upper bound, because its log-of-mean estimator is biased downward.
• The gap between block MDMs and the ARM is large at NFE=1 (about 80 PPL on OWT, L'=4) and narrows substantially only as NFE grows.

Limitations

• Empirical study limited to block MDMs; applying TUBE to non-block MDMs at long sequence lengths is considered challenging because the surrogate must approximate a vanishingly small probability.
• Tightness depends on the surrogate psi approximating p_model; the ARM surrogate leaves a much wider gap than the self-surrogate.
• Evaluation uses small block sizes (4, 8, 16) and uses LM1B models trained from scratch by the authors.

Experimental setup

Datasets: OWT LM1B

Benchmarks: OWT LM1B

Metrics: perplexity log-likelihood bound tightness

Baselines: CUBO TVO IS-VG-B ELBO ELBO_K ARM

Builds on: ELBO_K (I-WAE) CUBO TVO IS-VG-B BD3-LM

Closest papers by meaning

• Breaking AR’s Sampling Bottleneck: Provable Acceleration via Diffusion Language Models (NeurIPS 2025 · similarity 0.89)
• Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models (ICLR 2025 · similarity 0.89)
• Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data (ICLR 2025 · similarity 0.88)
• Diffusion Tree Sampling: Scalable inference‑time alignment of diffusion models (NeurIPS 2025 · similarity 0.88)
• Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling (ICLR 2025 · similarity 0.88)
• Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (ICML 2024 · similarity 0.88)