Research map / Vision-language and multimodal / Multimodal LLMs
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Show-o is a single 1.3B transformer (initialized from Phi-1.5) that unifies autoregressive next-token prediction for text with masked-token (discrete diffusion, MaskGIT-style) prediction for image tokens, handling understanding, text-to-image generation, inpainting, and mixed-modality tasks.
Problem
Existing unified multimodal systems pair separate understanding and generation models, or use fully autoregressive image generation, which needs many sampling steps; the paper asks whether one transformer can unify autoregressive and diffusion modeling for both understanding and generation.
Approach
Build on a pre-trained LLM with discrete MAGVIT-v2 image tokens in a shared vocabulary, using causal attention for text and full (omni) attention for image tokens, trained jointly with next-token prediction for text and MaskGIT-style mask token prediction for images, eliminating a separate text encoder and denoising network.
Key results
- Show-o (1.3B) reaches 80.0 POPE, 1097.2 MME, 69.4 VQAv2 and 62.5 Flickr30k, comparable to LLaVA-v1.5-Phi-1.5 (1.3B) and better than NExT-GPT-13B on VQAv2 (66.7).
- On GenEval, Show-o (1.3B) achieves 0.68 overall, above SDXL (0.55) and DALL-E 2 (0.52), and near SD3 (2B, 0.62).
- On zero-shot MSCOCO 30K, Show-o achieves FID 9.24 with 1.3B parameters and 35M training images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace discrete MAGVIT-v2 image tokens with CLIP-ViT continuous representations (ablation) | Show-o · Multimodal understanding benchmarks | POPE: 73.8 -> 84.5 | Helped |
| Replace discrete MAGVIT-v2 image tokens with CLIP-ViT continuous representations (ablation) | Show-o · Multimodal understanding benchmarks | Flickr30k: 36.2 -> 64.3 | Helped |
| Replace discrete MAGVIT-v2 image tokens with CLIP-ViT continuous representations (ablation) | Show-o · Multimodal understanding benchmarks | MMMU: 25.1 -> 27.4 | Mixed |
| Remove unified pretraining (ablation) | Show-o with discrete image tokens · Multimodal understanding benchmarks | Flickr30k: 36.2 -> 4.5 | Hurt |
| Remove unified pretraining (ablation) | Show-o with discrete image tokens · Multimodal understanding benchmarks | MMMU: 25.1 -> 28.1 | Mixed |
| Remove unified pretraining (ablation) | Show-o with CLIP-ViT continuous representations · Multimodal understanding benchmarks | Flickr30k: 64.3 -> 68.5 | Hurt |
| Use CLIP-ViT instead of MAGVIT-v2 continuous representations (ablation) | Show-o with unified pretraining · Multimodal understanding benchmarks | Flickr30k: 33.9 -> 64.3 | Helped |
| Increase image-text pretraining data from 35M to 2.0B (ablation) | Show-o, 256² input · Multimodal understanding | Flickr30k: 36.2 -> 48.9 | Helped |
| Increase image-text pretraining data from 35M to 2.0B (ablation) | Show-o, 256² input · Multimodal understanding | MMMU: 25.1 -> 25.0 | No clear effect |
| Increase input resolution from 256² to 512² (ablation) | Show-o trained on 2.0B image-text pairs · Multimodal understanding | Flickr30k: 48.9 -> 62.5 | Helped |
| Increase input resolution from 256² to 512² (ablation) | Show-o trained on 2.0B image-text pairs · Multimodal understanding | MMMU: 25.0 -> 26.7 | Helped |
| Unify multimodal understanding and generation in Show-o instead of the understanding-only LLaVA baseline | Phi-1.5 · GQA | score: 56.5 -> 58.0 | Helped |
Surprising findings
- Without unified pre-training, Show-o's discrete-token understanding collapses (e.g., Flickr30k 4.5 and POPE 63.8), showing that image-token embeddings must be learned from scratch with large-scale alignment.
- Continuous CLIP-ViT features give notably better understanding than discrete MAGVIT-v2 tokens, attributed to CLIP's larger pre-training data and discriminative objective.
Limitations
- FID on MSCOCO may not comprehensively reflect generation fidelity, as noted by the authors.
- Discrete image tokens from MAGVIT-v2 give weaker multimodal understanding than continuous CLIP features, and understanding performance lags the larger specialized models on some benchmarks (e.g., MMMU 26.7 vs 35.4 for LLaVA-v1.5 7B).
Experimental setup
Datasets: RefinedWeb ImageNet-1K CC12M SA1B LAION-aesthetics-12M DataComp COYO700M ShareGPT4V re-captions LLaVA-Pretrain-558K LLaVA-v1.5-mix-665K GenHowTo internal high-quality image-text pairs
Benchmarks: POPE MME Flickr30k VQAv2 GQA MMMU MSCOCO 30K FID GenEval
Metrics: accuracy CIDEr FID-30K GenEval overall score
Baselines: LLaVA-v1.5-Phi-1.5 LLaVA-v1.5 InstructBLIP Qwen-VL-Chat mPLUG-Owl2 NExT-GPT SEED-X DreamLLM VILA-U Emu3 Chameleon SD3
Builds on: MaskGIT MAGVIT-v2 Phi-1.5 LLaVA-v1.5 classifier-free guidance
Closest papers by meaning
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.91)
- Show-o2: Improved Native Unified Multimodal Models (NeurIPS 2025 · similarity 0.90)
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (CVPR 2025 · similarity 0.90)
- MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.90)
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.90)
- Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.