Research map / Vision-language and multimodal / CLIP and zero-shot
Emu: Generative Pretraining in Multimodality
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Emu is a 14B multimodal autoregressive model that predicts the next text token or regresses the next visual embedding over interleaved image/video/text sequences, with a Causal Transformer and a fine-tuned Stable Diffusion decoder for image generation.
Problem
Existing large multimodal models predict only next text tokens with frozen vision encoders and no visual supervision, and underuse video as a source of interleaved image-text data; the goal is a unified generative model producing both images and text.
Approach
Unify image and text modeling in one autoregressive sequence: encode images with EVA-CLIP, compress them into causal latent embeddings via a Causal Transformer, train with cross-entropy on text and l2 regression on visual embeddings, and decode images with a diffusion model conditioned on the regressed embeddings. Videos are converted into interleaved storyboard frames and subtitles (YT-Storyboard
Key results
- Emu achieves zero-shot COCO captioning CIDEr of 112.4, and Emu-I reaches 120.4, outperforming Flamingo-9B and Kosmos-1 by a large margin.
- Emu-I-14B scores 36.3 on MM-Vet, outperforming LLaVA-65B (35.5) and DreamLLM-7B (35.9).
- Zero-shot text-to-image FID on MS-COCO is 11.66 for Emu, better than GILL (12.20) but worse than SDv1.5 (9.93).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add YT-Storyboard-1B to pretraining (ablation) | Emu-7B · MS-COCO captioning | CIDEr: 110.8 -> 112.9 | Helped |
| add YT-Storyboard-1B to pretraining (ablation) | Emu-7B · OKVQA, zero-shot prompt with two text-only examples | VQA accuracy: 43.9 -> 42.3 | Hurt |
| add YT-Storyboard-1B to pretraining (ablation) | Emu-7B · OKVQA, 4-shot | VQA accuracy: 44.6 -> 45.7 | Helped |
| add YT-Storyboard-1B to pretraining (ablation) | Emu-7B · MSVDQA, zero-shot prompt with two text-only examples | Top-1 accuracy: 30.2 -> 30.8 | Helped |
| add YT-Storyboard-1B to pretraining (ablation) | Emu-7B · MSVDQA, 4-shot | Top-1 accuracy: 31.1 -> 34.9 | Helped |
| add YT-Storyboard-1B to pretraining (ablation) | Emu-7B · MSRVTTQA, zero-shot prompt with two text-only examples | Top-1 accuracy: 16.9 -> 17.9 | Helped |
| add YT-Storyboard-1B to pretraining (ablation) | Emu-7B · MSRVTTQA, 4-shot | Top-1 accuracy: 19.9 -> 20.8 | Helped |
| instruction-tune Emu with multimodal instruction data | Emu · VQAv2 | VQA accuracy: 52.0 -> 57.2 | Helped |
| instruction-tune Emu with multimodal instruction data | Emu · NExTQA | WUPS: 19.6 -> 5.8 | Hurt |
| add two text-only task examples to the prompt | Emu-I · VQAv2 | VQA accuracy: 57.2 -> 62.0 | Helped |
| add two text-only task examples to the prompt | Emu-I · VizWiz | VQA accuracy: 32.2 -> 38.3 | Helped |
| add multimodal Chain-of-Thought prompting by generating a caption first | Emu · multimodal understanding benchmarks | benchmark performance | Helped |
Surprising findings
- Emu-I (14B) outperforms the much larger Flamingo-80B on several VQA tasks such as VQAv2 and VizWiz.
- Image generation via regressed visual embeddings is inferior to SDv1.5 in FID, attributed to the condition-space mismatch and short decoder training.
Limitations
- Inherits hallucination, slow autoregressive inference, and outdated knowledge from LLMs/LMMs.
- Predominantly English-language training data; weaker multilingual ability.
- Image generation FID lags behind Stable Diffusion v1.5.
Experimental setup
Datasets: LAION-2B LAION-COCO MMC4 WebVid-10M YT-Storyboard-1B LAION-Aesthetics ShareGPT Alpaca LLaVA VideoChat-11K Video-ChatGPT-100k
Benchmarks: COCO Caption NoCaps Flickr30K VQAv2 OKVQA VizWiz VisDial MSVDQA MSRVTTQA NExTQA MS-COCO text2image FID MM-Vet
Metrics: CIDEr VQA accuracy NDCG Top-1 accuracy WUPS FID MM-Vet score
Baselines: Flamingo-9B Kosmos-1 MetaLM PALI-X-55B GILL CM3Leon AnyMAL-13B LLaVA-65B DreamLLM-7B SDv1.5
Builds on: Flamingo EVA-CLIP LLaMA Stable Diffusion LoRA
Closest papers by meaning
- Generative Multimodal Models are In-Context Learners (CVPR 2024 · similarity 0.94)
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.89)
- A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.89)
- Context-Aware Multimodal Pretraining (CVPR 2025 · similarity 0.88)
- LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.88)
- DuoGen: Towards Autonomous Interleaved Multimodal Generation (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.