Research map / Image generation / Text-to-image diffusion
MarkovGen: Structured Prediction for Efficient Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MarkovGen, which replaces the last few sampling steps of the Muse text-to-image model with a learned fully-connected MRF inference layer, achieving 1.5x speedup and better image quality.
Problem
Iterative token-based text-to-image generation (e.g., Muse) needs many sampling steps with large Transformers to make image tokens spatially and label-wise compatible, which is computationally expensive.
Approach
Model the VQGAN token grid jointly as a fully-connected MRF with learned spatial and label compatibility matrices, solved by differentiable mean-field inference, and use it to fast-forward the final sampling steps of Muse.
Key results
- MarkovGen achieves 1.5x speedup over full Muse (281 ms vs 442 ms per image on TPUv4).
- On MS-COCO 256x256, MarkovGen (18 iters) reaches FID 12.28 vs 13.13 for full Muse (24 iters) and 14.37 for early-exit Muse.
- Human raters strongly preferred MarkovGen outputs over both early-exit Muse and full Muse on 1633 Parti prompts (3 raters each).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace final Muse sampling steps with MRF inference | Muse base · MS-COCO, 256 × 256 | FID: 14.37 -> 12.28 | Helped |
| Replace final Muse sampling steps with MRF inference | Muse · Image generation inference | Time (ms): 442.05 -> 281.03 | Helped |
| Stop Muse iterations early without applying the MRF | Muse · MS-COCO, 256 × 256 | FID: 13.13 -> 14.37 | Hurt |
Surprising findings
- MRF-refined outputs with fewer steps beat the full 24-step Muse model in FID and human preference.
Limitations
- MRF does not use the text prompt; text guidance comes only through unaries
- Muse model was not trained jointly with the MRF layers
- Muse model was provided by the authors and not publicly released
Experimental setup
Datasets: WebLI
Benchmarks: MS-COCO (FID, 256x256) Parti Prompts (human side-by-side preference)
Metrics: FID human preference rate inference time (ms)
Baselines: Muse (full) Early exit Muse
Builds on: Muse fully-connected CRF VQGAN mean-field inference
Closest papers by meaning
- Rich Human Feedback for Text-to-Image Generation (CVPR 2024 · similarity 0.88)
- CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation (ICML 2025 · similarity 0.87)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.87)
- Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion (ICML 2024 · similarity 0.87)
- SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation (CVPR 2025 · similarity 0.86)
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.