Research map / Image generation / Fast diffusion sampling
Learnable Sampler Distillation for Discrete Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes learnable sampler distillation (LSD, and LSD+ with learned time schedules) that trains a few-step discrete diffusion sampler by aligning its intermediate concrete-score trajectory with a fine-step teacher, via learnable per-step coefficients.
Problem
Discrete diffusion models need many sampling steps; reducing steps amplifies compounding decoding error and discretization error, degrading sample quality.
Approach
Instead of matching non-differentiable final discrete samples, the student sampler learns per-step score-scaling coefficients by matching its intermediate scores to a high-step teacher's scores, with a Hamming-ball relaxed input objective; LSD+ additionally learns non-uniform step sizes via effective transition term matching.
Key results
- On SEDD-small at 8 NFEs, LSD+-Euler reaches generative perplexity 128.4 versus 308.1 for JYS-Euler and 423.1 for Euler.
- On RADD at 8 NFEs, LSD+-Euler achieves perplexity 89.8 versus 670.98 for vanilla Euler.
- On ImageNet 256x256 with MaskGIT at 4 NFEs, LSD+-Halton improves FID from 14.16 to 12.78.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LSD+ instead of the strongest reported baseline sampler | SEDD-small · SEDD-small text generation, 8–64 NFEs | generative perplexity: 307.382 -> 128.413 (8 NFEs); 125.283 -> 51.769 (16 NFEs); 55.842 -> 31.235 (32 NFEs); 31.192 -> 20.473 (64 NFEs) | Helped |
| Use LSD+ instead of the strongest reported baseline sampler | SEDD-medium · SEDD-medium text generation, 8–64 NFEs | generative perplexity: 299.394 -> 90.033 (8 NFEs); 115.853 -> 46.145 (16 NFEs); 43.958 -> 25.144 (32 NFEs); 25.545 -> 15.929 (64 NFEs) | Helped |
| Use LSD+ instead of the strongest reported baseline sampler | RADD · RADD text generation, 8–64 NFEs | generative perplexity: 648.736 -> 89.830 (8 NFEs); 123.585 -> 36.106 (16 NFEs); 89.912 -> 33.234 (32 NFEs); 66.549 -> 24.312 (64 NFEs) | Helped |
| Apply LSD+ to the Halton sampler | MaskGIT · ImageNet 256x256, 4 NFEs | FID: 14.16 -> 12.78 | Helped |
| Add the relaxed objective during training (ablation) | LSD+ with Euler on RADD · RADD, 8–64 NFEs | generative perplexity: 95.943 -> 89.830 (8 NFEs); 38.192 -> 36.106 (16 NFEs); 34.983 -> 33.234 (32 NFEs); 31.392 -> 29.115 (64 NFEs) | Helped |
| Set the Hamming distance threshold to 5% rather than 0% or 1% (ablation) | LSD+ on SEDD-small with Euler · SEDD-small, 32 inference steps | Perplexity: 35.98 -> 31.24 (0%); 32.15 -> 31.24 (1%) | Helped |
| Set the Hamming distance threshold to 10% or 20% instead of 5% (ablation) | LSD+ on SEDD-small with Euler · SEDD-small, 32 inference steps | Perplexity: 31.24 -> 39.97 (10%); 31.24 -> 51.52 (20%) | Hurt |
| Learn non-uniform time schedules in addition to sampler coefficients (ablation) | LSD, across text-generation backbones · Text generation | generative perplexity | Mixed |
Limitations
- Student performance is capped by the teacher sampler quality.
- No theoretical guarantees on distributional discrepancy between student and teacher in discrete space.
- No error bars or statistical significance reported.
Experimental setup
Datasets: OpenWebText CIFAR-10 ImageNet 256x256 Synthetic countdown task
Benchmarks: OpenWebText generative perplexity (GPT2-large) CIFAR-10 FID ImageNet 256x256 FID synthetic countdown error rate GSM8K MATH
Metrics: generative perplexity FID error rate MAUVE entropy accuracy throughput
Baselines: Euler Tweedie tau-leaping JYS-Euler JYS-Tweedie theta-RK2 theta-trapezoidal CTMC Halton tauLDR-10 ReMDM Fast-dLLM
Builds on: S4S LD3 JYS tau-leaping SEDD
Closest papers by meaning
- Distilling ODE Solvers of Diffusion Models into Smaller Steps (CVPR 2024 · similarity 0.90)
- Jump Your Steps: Optimizing Sampling Schedule of Discrete Diffusion Models (ICLR 2025 · similarity 0.90)
- Simple and Fast Distillation of Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Informed Correctors for Discrete Diffusion Models (NeurIPS 2025 · similarity 0.89)
- Align Your Steps: Optimizing Sampling Schedules in Diffusion Models (ICML 2024 · similarity 0.89)
- SCott: Accelerating Diffusion Models with Stochastic Consistency Distillation (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.