Research map / Vision-language and multimodal / Multimodal LLMs
Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Discrete Diffusion Timestep (DDT) tokens, recursive discrete visual tokens learned by a diffusion-decoder tokenizer, and uses them in DDT-LLaMA, an 8B MLLM that unifies image understanding and generation.
Problem
Existing unified MLLMs use spatial visual tokens (raster-ordered image patches), which lack the recursive structure of language and so are hard for an LLM to model, limiting joint comprehension and generation.
Approach
Learn a 1D sequence of discrete tokens where each token appended over diffusion timesteps compensates for attribute loss in noisy images; a diffusion decoder (MMDiT) reconstructs the clean image from the first t tokens, and the resulting tokens are modeled autoregressively by an LLM alongside text.
Key results
- On GenEval overall, DDT-LLaMA (8B) scores 0.66, above SEED-X (0.49), Emu3 (0.54) and SDXL (0.55).
- On MLLM comprehension benchmarks DDT-LLaMA reports 124.2 NoCaps CIDEr, 66.3 GQA, and 1378.9 MME, exceeding the listed 8B unified baselines on most benchmarks.
- Tokenizer trained only on ImageNet reaches 21.7 PSNR on ImageNet validation and class-conditional FID 6.1 with a GPT model, better than MoVQ (7.1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DDT-LLaMA instead of SD3 | zero-shot text-to-image generation · GenEval | Overall: 0.74 -> 0.66 | Hurt |
| Use DDT-LLaMA instead of DALL-E 3 | zero-shot text-to-image generation · T2I-CompBench | Color: 0.811 -> 0.728 | Hurt |
| Use DDT-LLaMA instead of the strongest reported MLLM baseline | zero-shot image editing · EVR | CVS: 81.9 -> 85.5 | Helped |
| Use DDT-LLaMA instead of MGIE | zero-shot image editing · MA5K | LPIPS: 29.8 -> 29.6 | Helped |
| Use DDT-LLaMA instead of Emu2-Chat | vision-language comprehension · GQA | accuracy: 65.1 -> 66.3 | Helped |
| Use DDT-LLaMA instead of Emu2-Chat | vision-language comprehension · VQAv2 | accuracy: 84.9 -> 76.4 | Hurt |
| Use DDT tokens instead of MoVQ spatial tokens (ablation) | Gemma2-2B multimodal model · image editing A/B test | cases preferred: 10 -> 65 | Helped |
| Scale the backbone from 2B to 8B (ablation) | DDT-based MLLM · image editing | editing performance | Helped |
| Use DDT tokens rather than VQGAN tokens (ablation) | multimodal autoregressive pretraining · text generation | perplexity | Helped |
| Increase model size and training compute (ablation) | DDT-based MLLM · text-to-image generation | visual quality | Helped |
Surprising findings
- Spatial token sequences are largely insensitive to order during LLM training, unlike language, while DDT tokens behave like language under perturbation.
- Perplexity of text generation decreases when training with DDT tokens but increases with VQGAN tokens.
Limitations
- Tokenizer trained only on ImageNet at 256x256 reconstructs open-domain images worse than baselines like MoVQ, limiting aesthetic quality.
- Underperforms on the TwoObj GenEval subtask because training captions rarely use that prompt format.
- Authors state that code, models and a full technical report are forthcoming, so results are not yet reproducible.
Experimental setup
Datasets: ImageNet LAION COYO ShareGPT4V ALLaVA GRIT InstructPix2Pix Hive InstructBLIP datasets Wikipedia Pile
Benchmarks: GenEval T2I-CompBench DrawBench EVR MA5K MagicBrush NoCaps Flickr30K VQAv2 GQA OKVQA VizWiz
Metrics: GenEval overall score T2I-CompBench scores CLIP-T L1 CVS LPIPS CIDEr VQA accuracy MME perception SEED-Bench accuracy
Baselines: SDXL DALL-E 3 SD3 PixArt-alpha SEED-X Emu3 Lumina-mGPT VILA-U LaVIT SEED-LLaMA Emu2-Gen InsPix2Pix
Builds on: LLaMA-3 MMDiT Rectified Flow Stable Diffusion 3 VQ-VAE
Closest papers by meaning
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.93)
- Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents (NeurIPS 2025 · similarity 0.91)
- Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Models (CVPR 2026 · similarity 0.91)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.91)
- I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models (ICML 2025 · similarity 0.90)
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.