Research map / Image generation / Autoregressive and latent generation
Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Discrete Latent Codes (DLCs), sequences of discrete tokens derived from SSL-trained Simplicial Embeddings on DINOv2, to condition diffusion models, generating them with a discrete diffusion model (SEDD) and enabling compositional and text-to-DLC image generation.
Problem
Diffusion models struggle to fit highly multimodal continuous distributions such as unconditional ImageNet, and continuous SSL embeddings used for conditioning are hard to sample from and not flexibly compositional.
Approach
Represent images as sequences of discrete tokens (argmax over simplicial embeddings of a finetuned DINOv2), model the code distribution p(c) with a discrete diffusion model, and model p(x|c) with a DiT, so that p(x) factorizes into two easier models; token-level sampling enables compositional recombination of two images.
Key results
- DiT-XL/2 with DLC_512 reaches unconditional ImageNet FID 2.00 without CFG, versus RCG's 3.44 and MAR-H's 2.35 reported in the table.
- With CFG, unconditional DLC_512 achieves FID 1.59, while label-conditioned REPA achieves 1.42.
- On composed image generation, DLC_512 achieves Vendi score 13.2 versus 9.6 for DINOv2 and 4.2 for RCG.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition DiT-XL/2 on DLC₅₁₂ instead of class labels | DiT-XL/2 · ImageNet | FID | Helped |
| Use DLC₅₁₂ rather than RCG conditioning | DiT-XL/2 · ImageNet | Generative FID: 4.89 -> 4.21 | Helped |
| Use DLC₅₁₂ rather than DINOv2 continuous embeddings | DiT-XL/2 · ImageNet | Generative FID: 37.90 -> 4.21 | Helped |
| Add classifier-free guidance | DLC₅₁₂ unconditional image generation · ImageNet | FID: 2.00 -> 1.59 | Helped |
| Compose two source representations by randomly selecting each token position from either DLC | DLC-conditioned image generation · Composed image samples | Vendi score: 9.6± 1.3 -> 13.2± 0.8; 4.2± 0.9 -> 13.2± 0.8 | Helped |
| Increase DLC sequence length (ablation) | DLC image and code generators · ImageNet | FID | Helped |
| Vary the remasking ratio η (ablation) | SEDD-Absorb DLC sampler · ImageNet 256×256 | FID | Mixed |
Surprising findings
- Unconditional DiT-XL/2 with DINOv2 embeddings given ground-truth (encoded) embeddings reaches much lower FID than with generated embeddings (2.12 enc-dec vs 37.90 gen FID), suggesting the failure lies in modeling the embedding distribution.
- Remasking may be usable as a replacement for classifier-free guidance in DLC sampling, according to the appendix.
Limitations
- Text-to-image pipeline relies on frozen DiT and is only a proof of concept with 9M pairs and qualitative evaluation.
- Longer DLC sequences converge more slowly and are more expensive to model with discrete diffusion.
- Novelty evaluation is largely qualitative via nearest-neighbour retrieval.
Experimental setup
Datasets: ImageNet-1K LAION (9M subset)
Benchmarks: ImageNet
Metrics: FID sFID IS precision recall linear probe accuracy Vendi score
Baselines: RCG MAR-H SiT-XL/2 REPA DiT-XL/2 DisCo-Diff Mage-L Stable Diffusion
Builds on: Simplicial Embeddings DINOv2 DiT SEDD REPA LLaDA
Closest papers by meaning
- DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents (ICML 2024 · similarity 0.90)
- Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.90)
- Boosting Generative Image Modeling via Joint Image-Feature Synthesis (NeurIPS 2025 · similarity 0.90)
- Nested Diffusion Models Using Hierarchical Latent Priors (CVPR 2025 · similarity 0.90)
- On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.