Research map / Vision-language and multimodal / Human and embodied modeling
Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Transfusion trains a single transformer on mixed text and image sequences with next-token loss for text and diffusion loss for image patches, showing better scaling than Chameleon-style discrete image tokens.
Problem
Training a single multi-modal model that generates both discrete text and continuous images without losing information from quantizing images into discrete tokens.
Approach
Combine language modeling loss on text with DDPM diffusion loss on image latent patches over shared transformer parameters, using causal attention across the sequence and bidirectional attention within each image, with optional U-Net patch encoders/decoders.
Key results
- Transfusion 7B reaches FID 16.8 vs Chameleon 29.6 under controlled 0.5T-token training, with FID parity at 34x less compute (parity FLOP ratio 0.029).
- A 7B Transfusion model trained on 2T tokens scores 0.63 on GenEval, above DALL-E 2 (0.52) and SDXL (0.55), and 66.1 on Llama 2 eval suite, on par with Llama 1 and 2.
- U-Net patch encoding with 2x2 patches improves 0.76B FID from 20.3 (linear) to 16.7, and compressing images to 16 patches costs modest performance with linear encoding.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use continuous image diffusion instead of quantized image tokens | Transfusion vs Chameleon · MS-COCO 30k | FID: 29.6 -> 16.8 | Helped |
| use continuous image diffusion instead of quantized image tokens | Transfusion vs Chameleon · C4 | perplexity: 8.41 -> 7.72 | Helped |
| use continuous image diffusion instead of quantized image tokens | Transfusion vs Chameleon · MS-COCO 5k | CIDEr: 18.0 -> 27.2 | Helped |
| use bidirectional instead of causal intra-image attention (ablation) | Transfusion with linear image encoding · MS-COCO 30k | FID: 61.3 -> 20.3 | Helped |
| use bidirectional instead of causal intra-image attention (ablation) | Transfusion with U-Net image encoding · MS-COCO 30k | FID: 16.8 -> 16.7 | No clear effect |
| increase linear patch size from 2×2 to 8×8 latent pixels (ablation) | Transfusion 0.76B · MS-COCO 30k | FID: 20.3 -> 43.5 | Hurt |
| increase U-Net patch size from 2×2 to 4×4 latent pixels (ablation) | Transfusion 0.76B · MS-COCO 5k | CIDEr: 25.4 -> 29.9 | Helped |
| increase U-Net patch size from 2×2 to 8×8 latent pixels (ablation) | Transfusion 0.76B · MS-COCO 5k | CIDEr: 25.4 -> 29.5 | Helped |
| replace linear image encoder/decoder with U-Net (ablation) | Transfusion 0.76B · MS-COCO 5k | CIDEr: 16.0 -> 25.4 | Helped |
| replace linear image encoder/decoder with U-Net (ablation) | Transfusion 0.76B · MS-COCO 30k | FID: 20.3 -> 16.7 | Helped |
| limit diffusion noise to maximum t=500 when images precede captions (ablation) | Transfusion 0.76B · MS-COCO 5k | CIDEr: 25.4 -> 29.4 | Helped |
| limit diffusion noise to maximum t=500 when images precede captions (ablation) | Transfusion 7B · MS-COCO 5k | CIDEr: 33.7 -> 35.2 | Helped |
Surprising findings
- Transfusion also improves text-only perplexity over Chameleon at matched FLOPs, even though both model text identically.
- Causal-only attention within images hurts text-to-image FID substantially with linear encoders (61.3 vs 20.3 with bidirectional).
Limitations
- Image data is largely proprietary (Shutterstock) and the 7B model is not released.
- Image editing results are only qualitative after fine-tuning on 8k examples.
- Lambda coefficient and U-Net scaling were not tuned beyond preliminary experiments.
Experimental setup
Datasets: Llama 2 text corpus Shutterstock licensed images publicly available image-caption data CC12M MS-COCO EmuEdit image editing examples (8k subset used for fine-tuning) Wikipedia C4
Benchmarks: Wikipedia perplexity C4 perplexity Llama 2 eval suite MS-COCO 5k CIDEr MS-COCO 30k FID MS-COCO 30k CLIP GenEval
Metrics: perplexity accuracy CIDEr FID CLIP score GenEval score parity FLOP ratio
Baselines: Chameleon Llama 2 DALL-E 2 SDXL DeepFloyd SD 3 Imagen Parti
Builds on: DDPM latent diffusion models Llama 2 Chameleon classifier-free guidance
Closest papers by meaning
- LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding (CVPR 2026 · similarity 0.86)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.86)
- CapsFusion: Rethinking Image-Text Data at Scale (CVPR 2024 · similarity 0.86)
- One-Way Ticket: Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.85)
- Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (NeurIPS 2024 · similarity 0.85)
- Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.