Research map / Image generation / Fast diffusion sampling
ShortListing Model: A Streamlined Simplex Diffusion for Discrete Variable Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Shortlisting Models (SLM), a simplex-based discrete diffusion that prunes candidate sets from all categories down to one-hot, trained with a cross-entropy-style objective, and evaluated on text8, OpenWebText, DNA and protein design.
Problem
Generative modeling of discrete variables (text, DNA, protein sequences) where existing simplex-based diffusion/flow methods rely on complex geodesics or vocabulary-level MSE losses that suffer gradient vanishing and scale poorly to large vocabularies.
Approach
Model discrete generation as progressive candidate pruning: a Bernoulli forward process appends candidates until all categories are included, and the reverse process transitions among simplex centroids (candidate subspaces), trained with a simplified cross-entropy objective instead of Bernoulli KL/MSE.
Key results
- On text8, SLM (L_weight) reaches BPC ≤1.38, the best among non-autoregressive methods reported (vs. MDLM ≤1.39, SFM 1.39).
- On promoter design, SLM achieves MSE 0.0265±0.0006, lower than MDLM (0.028) and Fisher-Flow (0.029); with CFG, SLM enhancer FBD is 1.4 on Mel and 1.0 on FB.
- On OpenWebText, SLM (110M) gets PPL 53.90 and Gen-PPL 65.59 versus BFN (110M) PPL 105.66 and Gen-PPL 299.95, but still lags AR and MDLM in likelihood.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use reweighted loss instead of simple loss (ablation) | SLM · Text8 test set | BPC: $\leq$ 1.42 -> $\leq$ 1.38 | Helped |
| Reduce depth and increase width | SLM (110M) · OpenWebText validation | PPL: 53.90 -> 43.25 | Helped |
| Reduce depth and increase width | SLM (110M) · OpenWebText validation | Gen-PPL: 65.59 -> 53.79 | Helped |
| Use classifier-free guidance | SLM · Enhancer fly brain | FB FBD: 4.4$\pm$0.2 -> 1.0$\pm$0.1 | Helped |
| Use classifier-free guidance | SLM · Enhancer melanoma | Mel FBD: 2.2$\pm$0.1 -> 1.4$\pm$0.1 | Helped |
| Use simple loss instead of reweighted loss (ablation) | SLM · Enhancer melanoma | FBD: 2.5848 -> 2.1788 | Helped |
| Use simple loss instead of reweighted loss (ablation) | SLM · Enhancer fly brain | FBD: 4.9670 -> 4.4450 | Helped |
| Use reweighted loss instead of simple loss (ablation) | SLM · Promoter design | MSE: 0.0265 -> 0.0260 | Helped |
| Use reweighted loss instead of simple loss (ablation) | SLM · Promoter design | PPL: 2.8084 -> 2.7672 | Helped |
| Increase protein sampling steps (ablation) | SLM · Protein design | pLDDT | Helped |
| Increase protein sampling steps (ablation) | SLM · Protein design | Progen2-nll | Helped |
Surprising findings
- Simplex-based methods struggle with large vocabularies because an H-dimensional embedding cannot represent K orthogonal token vectors; widening the network (SLM_W) improves OpenWebText PPL from 53.90 to 43.25 at similar parameter count.
- The simplified unweighted cross-entropy loss yields better generated samples while the reweighted ELBO-aligned loss yields better likelihood, a trade-off.
Limitations
- Likelihood (PPL/BPC) on OpenWebText still lags autoregressive and MDLM baselines.
- Evaluation limited to text and biological sequences; graphs and images are left for future work.
- Scaling to large real-world applications needs further theoretical analysis and engineering improvements.
Experimental setup
Datasets: text8 OpenWebText Human Promoter Database CAGE signal data fly brain enhancer data human melanoma enhancer data UniRef50
Benchmarks: text8 OpenWebText Human Promoter Database DNA design enhancer FBD (Mel, FB) UniRef50 protein design MNIST (dynamically binarized)
Metrics: BPC PPL Gen-PPL Entropy MSE FBD pLDDT Progen2 NLL scPerplexity inner-TM
Baselines: Transformer AR MDLM SEDD UDLM D3PM BFN SFM DFM Fisher-Flow EvoDiff ESM2-150M DDSM
Builds on: MDLM Dirichlet Flow Matching Bayesian Flow Networks DDSM EvoDiff
Closest papers by meaning
- Diffusion Language Models Are Versatile Protein Learners (ICML 2024 · similarity 0.86)
- Diffusion on Language Model Encodings for Protein Sequence Generation (ICML 2025 · similarity 0.85)
- Dirichlet Flow Matching with Applications to DNA Sequence Design (ICML 2024 · similarity 0.84)
- Bio-xLSTM: Generative modeling, representation and in-context learning of biological and chemical sequences (ICLR 2025 · similarity 0.84)
- Absorb & Escape: Overcoming Single Model Limitations in Generating Heterogeneous Genomic Sequences (NeurIPS 2024 · similarity 0.83)
- Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.