Research map / Image generation / Text-to-image diffusion
Muse: Text-To-Image Generation via Masked Generative Transformers
Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Patrick Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, Dilip Krishnan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use of parallel iterative decoding with cosine schedule (24 steps base, 8 steps super-res) instead of per-token autoregressive decoding | Muse inference · Inference speed (TPUv4) | per-batch inference time: Muse-3B 256x256 0.5s vs Parti-3B 6.4s | Improved |
| Super-resolution stage added on top of base model (base + super-res vs base alone) (ablation) | Muse on CC3M · CC3M | FID: 6.8 -> 6.06 | Improved |
| Scaling Muse to 3B parameters with T5-XXL conditioning, zero-shot on COCO | Muse · MS-COCO zero-shot 256x256 | FID-30K: 7.88 vs Parti-20B 7.23 and Imagen 7.27 | Mixed |
| Frozen pre-trained T5-XXL LLM text embeddings as conditioning (vs learning language model from scratch) (ablation) | Muse · Qualitative/initial experiments | performance | Improved |
| Linearly increasing classifier-free guidance scale during iterative sampling | Muse sampling · CFG trade-off | diversity vs fidelity | No clear effect |
| Decoder finetuning with more residual blocks and channels while keeping encoder and transformers frozen (ablation) | VQGAN decoder · Qualitative reconstruction | visual detail | Improved |
| Muse-3B vs Stable Diffusion v1.4 on PartiPrompts with human raters | Muse-3B · PartiPrompts human eval | prompt alignment preference: 70.6% vs 25.4% | Improved |
| Best-of-16 reranking by CLIP score per prompt | Muse generations for human eval and examples · PartiPrompts / qualitative | selection | No clear effect |
| Scaling the base model to 3B parameters with 48 layers | Muse base model · MS-COCO zero-shot | CLIP score: 0.32 vs Imagen 0.27 | Improved |
Later papers that compared with this method
1 later accepted paper reports Muse-3B as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths (NeurIPS 2023 · better on 1 of 1 shared result)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis (ICLR 2025 · similarity 0.92)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.91)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.90)
- JetFormer: An autoregressive generative model of raw images and text (ICLR 2025 · similarity 0.89)
- Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.89)
- Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
