Research map / Vision-language and multimodal / Multimodal LLMs
DreamLLM: Synergistic Multimodal Comprehension and Creation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamLLM, an MLLM that generates raw interleaved image-text documents end-to-end by using learnable dream queries to condition a frozen Stable Diffusion decoder via score distillation, unifying comprehension and creation.
Problem
Existing MLLMs for image generation align outputs to CLIP embedding spaces, losing modality-specific information and failing to generate raw interleaved multimodal documents, so comprehension and creation do not synergize well.
Approach
Learnable dream queries appended to the MLLM context produce conditioning embeddings for a frozen Stable Diffusion decoder trained by denoising score matching (score distillation), so images are modeled in raw pixel space; a <dream> token predicts image placement, and interleaved generative pretraining (I-GPT) on web documents unifies all conditional, marginal and joint distributions under one aut
Key results
- DreamLLM-7B reaches 8.46 zero-shot FID on MS-COCO 30K, 3.20 FID better than Emu-13B.
- DreamLLM-7B scores 49.9 on MMBench and 35.9 on MM-Vet in zero-shot evaluation, state of the art among 7B models reported.
- In human evaluation of generated interleaved documents, 60.68% of samples were judged supported versus 77.24% for ground-truth MMC4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add interleaved generative pretraining (I-GPT) stage with MMC4 data (ID setting) (ablation) | DreamLLM · MS-COCO text-to-image and comprehension synergy analysis | FID / comprehension | Helped |
| Add CLIP alignment loss to DreamLLM training (ablation) | DreamLLM · Synergy ablation (text-to-image) | convergence | Hurt |
| Use 64 dream queries instead of 32 or 128 (ablation) | DreamLLM · MS-COCO 30K zero-shot FID | FID: 8.46 vs 9.56 (32) and 14.24 (128) | Helped |
| Stage-I alignment training of DreamLLM (vs. Stable Diffusion baseline) | DreamLLM · MS-COCO and LN-COCO zero-shot FID | FID: reduces FID by 3.67 on MS-COCO and 11.83 on LN-COCO | Helped |
| Learned dream queries with MLLM conditioning instead of CLIP-aligned conditional embeddings | DreamLLM vs concurrent MLLM image synthesis (Emu) · MS-COCO zero-shot FID | FID: surpasses Emu-13B by 3.20 FID | Helped |
| Joint image-text training with Vicuna-based DreamLLM vs language-only LLM on language benchmarks | DreamLLM · Language benchmarks (PIQA, SIQA, HellaSwag, WinoGrande, BoolQ) | accuracy: +1.0 to +1.7 on commonsense/reading, -3.2 on MMLU | No clear effect |
Surprising findings
- Adding CLIP alignment loss to the objective caused training to collapse rather than converge.
- Relative to Stable Diffusion, DreamLLM adds only about 0.2s average inference latency.
Limitations
- Explicit CLIP alignment causes training collapse, so the approach relies on the score-distillation objective alone.
- Image consistency across generated images in interleaved documents is poor without further fine-tuning on subject-driven data.
- Human evaluation supportive rate (60.68%) remains below ground-truth MMC4 (77.24%).
Experimental setup
Datasets: MMC4-Core LAION-400M ShareGPT MS-COCO LN-COCO LLaVA-1.5 instruction data
Benchmarks: MS-COCO LN-COCO COCO captioning Image2Paragraph VQAv2 OKVQA VizWiz TextVQA MMBench MM-Vet POPE MMLU
Metrics: FID CIDEr accuracy FID on interleaved subset
Baselines: Emu-14B CM3Leon-7B Stable Diffusion LLaVA-7B Kosmos-1 Flamingo-9B
Builds on: Stable Diffusion Vicuna CLIP Emu Flamingo
Closest papers by meaning
- CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation (CVPR 2025 · similarity 0.90)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.89)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.89)
- WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (CVPR 2026 · similarity 0.88)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.88)
- MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.