Research map / Image generation / Fast diffusion sampling

DiffScore: Text Evaluation Beyond Autoregressive Likelihood

NeurIPS 2026 · Image generation › Fast diffusion sampling · approach: Masked diffusion language model decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DiffScore, a text evaluation framework that scores candidate text by masked-reconstruction ELBO of masked diffusion LLMs (LLaDA) instead of autoregressive likelihood, with zero-shot and LoRA fine-tuned variants.

Problem

Autoregressive likelihood-based text evaluation suffers from positional bias (early tokens scored with little context) and directional asymmetry (Reversal Curse), conflating architecture artifacts with text quality.

Approach

Score each token under bidirectional context via masked reconstruction, estimating the ELBO of a masked diffusion LM with Monte Carlo sampling over masking rates; the same mechanism yields multi-timestep quality profiles and a PMI decomposition separating fluency from faithfulness.

Key results

• DiffScore-FT average segment-level Kendall tau 0.356 vs BARTScore 0.342 on WMT19 (MT).
• DiffScore-FT average Spearman 0.544 vs BARTScore 0.519 across six summarization benchmarks.
• DiffScore-FT average system-level Spearman 0.310 vs BARTScore 0.240 on data-to-text; zero-shot DiffScore matches G-Eval average (0.245).

Surprising findings

• Zero-shot Dream-7B yields near-random correlations, and only fine-tuning unlocks its performance.
• G-Eval shows cross-lingual instability on WMT19 (tau 0.120 on fi-en vs 0.520 on ru-en).

Limitations

• Per-sample latency (0.88 s) and 8B backbone are much higher than BARTScore's cost.
• Zero-shot performance depends on pretraining scale; smaller or weaker MDLLM (Dream) is near random zero-shot.

Experimental setup

Datasets: CNN/DailyMail 3.0.0 SummEval Newsroom REALSumm Rank19 QAGS-CNN QAGS-XSUM WMT19 BAGEL SFHOT SFRES

Benchmarks: SummEval Newsroom REALSumm Rank19 QAGS-CNN QAGS-XSUM WMT19 (7 language pairs) BAGEL SFHOT SFRES

Metrics: Kendall tau Spearman rho Pearson r pairwise accuracy

Baselines: BLEU ROUGE METEOR BERTScore MoverScore BARTScore GPTScore UniEval AlignScore QuestEval G-Eval

Builds on: BARTScore LLaDA GPTScore G-Eval

Closest papers by meaning

• Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.87)
• Anchored Diffusion Language Model (NeurIPS 2025 · similarity 0.87)
• Simple and Effective Masked Diffusion Language Models (NeurIPS 2024 · similarity 0.87)
• Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.86)
• Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.86)
• Beyond Autoregression: Fast LLMs via Self-Distillation Through Time (ICLR 2025 · similarity 0.86)