Research map / Vision-language and multimodal / Human and embodied modeling
Dual Diffusion for Unified Image Generation and Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes D-DiT, a fully end-to-end dual-branch diffusion transformer initialized from SD3-medium that jointly trains flow-matching image generation and masked discrete text diffusion for captioning, VQA, and T2I.
Problem
Diffusion models excel at text-to-image generation but lack competitive visual understanding (captioning, VQA), an area dominated by autoregressive vision-language models; prior diffusion multimodal models rely on AR decoders or CLIP-latent text diffusion.
Approach
A single MM-DiT-style transformer with two branches is trained under one joint loss: continuous flow matching on image latents and absorbing-state masked diffusion on T5 text tokens, enabling image-to-text infilling for VQA alongside text-to-image generation.
Key results
- On VQA benchmarks, D-DiT (512x512, 2B) reaches 60.1 VQAv2, 1124.7 MME, 59.2 GQA, 84.0 POPE, outperforming Show-O 512 on MME, GQA, and POPE.
- On GenEval, D-DiT (2B) achieves overall 0.65, close to SD3 (0.62) and Show-O (0.68).
- Text-diffusion captioning on MS-COCO reaches CIDEr 56.2 at 512 resolution (Table 3), and D-DiT's text branch beats frozen CLIP ViT L/14 + GPT-2 on GPT-4-judged VQAv2 probing (55.0 vs 50.6 zero-shot).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase text diffusion sampling steps (ablation) | D-DiT · MS-COCO | CIDEr: 20.2 -> 56.2 | Helped |
| increase text diffusion sampling steps (ablation) | D-DiT · VQAv2 | accuracy: 58.8 -> 60.5 -> 59.6 | No clear effect |
| use trainable SD3 image features instead of frozen SD3 features (ablation) | SD3 feature + GPT2 · VQAv2 (val, 0-shot) | accuracy: 42.3 -> 45.1 | Helped |
| use CLIP ViT L/14 features instead of SD3 features (ablation) | image captioning model with GPT2 · VQAv2 (val, 0-shot) | accuracy: 45.1 -> 50.6 | Helped |
| use D-DiT text output instead of a separate GPT2 decoder (ablation) | image captioning model · VQAv2 (val, 0-shot) | accuracy: 50.6 -> 55.0 | Helped |
| use D-DiT text output instead of a separate GPT2 decoder (ablation) | image captioning model · VQAv2 (val, 32-shot) | accuracy: 54.8 -> 60.3 | Helped |
Surprising findings
- Diffusion features from SD3 used as a prefix to a GPT-2 decoder underperform CLIP ViT L/14, even when the diffusion backbone is unfrozen.
- The model reaches meaningful text output after fewer than 25B text tokens when initialized from SD3, and was never trained on text-only data.
Limitations
- Lags autoregressive VLMs such as QWEN-VL and BLIP-2 on VQA benchmarks, and the 256 resolution variant is notably weaker.
- Does not model unconditional text marginals; the model was never trained on text-only generation.
- Text generation requires many diffusion sampling steps (e.g. 256 for long responses), increasing inference cost.
Experimental setup
Datasets: DataComp-1B ShareGPT4V OpenImages LLaVA-Pretrain558K LLaVA-v1.5-mix-665K TextVQA VizWiz LAION-1024 Midjourney images
Benchmarks: VQAv2 VizWiz OKVQA GQA POPE MME MS-COCO GenEval
Metrics: accuracy CIDEr GenEval overall score
Baselines: Show-O Transfusion UniDiffuser Versatile Diffusion CLIP ViT L/14 + GPT-2 SD3 QWEN-VL BLIP-2 LLaVA-Next InternVL-2.0
Builds on: Stable Diffusion 3 MD4 masked diffusion language model UniDiffuser LLaVA-1.5
Closest papers by meaning
- Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens (CVPR 2025 · similarity 0.93)
- Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusion (ICML 2026 · similarity 0.92)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.92)
- I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models (ICML 2025 · similarity 0.92)
- Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces (ICML 2025 · similarity 0.92)
- Diffusion Models for Multi-Task Generative Modeling (ICLR 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.