Research map / Image generation / Fast diffusion sampling
Target Concrete Score Matching: A Holistic Framework for Discrete Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Target Concrete Score Matching (TCSM), a discrete diffusion training objective that matches concrete scores of the clean-data target distribution, unifying pre-training, reward/preference fine-tuning, and AR-to-diffusion distillation.
Problem
Discrete diffusion models are fragmented in their training objectives and lag autoregressive models in post-training (reward, preference, distillation), since existing objectives operate in noisy intermediate spaces.
Approach
Match the concrete score of the clean-data denoising distribution p_{1|t} (the target) rather than the noising-kernel concrete score, using a k-Hamming neighborhood and a generalized KL Bregman divergence; this operates in clean data space so reward functions and pretrained clean-data models plug in directly.
Key results
- On OpenWebText zero-shot perplexity, TCSM-Hollow variants and TCSM-distrib (e.g., 47.29 LAMBADA, 31.56 WikiText) beat MDLM (47.52, 32.83) in the reported table.
- TCSM-distrib with a masked source reaches 65.82 on 1BW versus MDLM's 67.01 in the reported table.
- Parametric target models (TCSM-AR, TCSM-Bert, TCSM-Hollow) train consistently faster than baselines according to validation NLL curves.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with TCSM-distrib instead of MDLM | Discrete diffusion language models · LAMBADA | perplexity: ≤ 47.52 -> ≤ 47.29 | Helped |
| Train with TCSM-distrib instead of MDLM | Discrete diffusion language models · PTB | perplexity: ≤ 95.26 -> ≤ 95.26 | No clear effect |
| Train with TCSM-distrib instead of MDLM | Discrete diffusion language models · WikiText | perplexity: ≤ 32.83 -> ≤ 31.56 | Helped |
| Train with TCSM-distrib instead of MDLM | Discrete diffusion language models · LM1B | perplexity: ≤ 67.01 -> ≤ 65.82 | Helped |
| Use generalized KL instead of BCE density-ratio estimation | TCSM post-training · OpenWebText validation | perplexity: 21.87 -> 21.74 | Helped |
| Use LSIF instead of BCE density-ratio estimation | TCSM post-training · OpenWebText validation | perplexity: 21.87 -> 22.10 | Hurt |
| Apply TCSM density-ratio post-training relative to a pretrained diffusion model | Pretrained discrete diffusion models · Language modeling | perplexity | Helped |
| Use a parametric target model p1 during TCSM training | Discrete diffusion models · OpenWebText | perplexity | Helped |
| Use pre-trained autoregressive-model target estimates | TCSM pre-training · OpenWebText | perplexity | Helped |
| Use pre-trained BERT target estimates | TCSM pre-training · OpenWebText | perplexity | Helped |
| Use a hollow transformer for parametric target estimation | TCSM pre-training · OpenWebText | perplexity | Helped |
| Use stronger preference optimization | TCSM preference fine-tuning · IMDB sentiment | mean reward | Helped |
Limitations
- Post-training experiments are limited to small settings (IMDB sentiment, toxicity, synthetic 2D grid).
- Distillation requires O(VL) teacher evaluations, approximated via top-K or first-order Taylor methods.
Experimental setup
Datasets: OpenWebText text8 IMDB
Benchmarks: LAMBADA WikiText Penn Treebank One Billion Words text8
Metrics: zero-shot perplexity bits-per-character validation NLL generative perplexity reward entropy
Baselines: SEDD MDLM MD4 D3PM Plaid EDLM GPT-2
Builds on: SEDD discrete flow matching MDLM MD4 concrete score matching
Closest papers by meaning
- Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (ICML 2024 · similarity 0.92)
- Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysis (ICLR 2025 · similarity 0.90)
- Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data (ICLR 2025 · similarity 0.89)
- Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models (ICLR 2025 · similarity 0.89)
- Diffusion Beats Autoregressive in Data-Constrained Settings (NeurIPS 2025 · similarity 0.89)
- Continuous Diffusion Model for Language Modeling (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.