Research map / Image generation / Text-to-image diffusion
Towards Understanding and Quantifying Uncertainty for Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces prompt-space uncertainty quantification for text-to-image models, adapting image-space OOD methods (DDPM-OOD, LMD, 2XDM) and proposing PUNC, which captions generated images with an LVLM and scores caption-prompt similarity via ROUGE/BERTScore precision and recall.
Problem
Quantifying how uncertain a text-to-image generative model is with respect to its text prompt, including disentangling aleatoric and epistemic uncertainty, which image-space methods cannot do well.
Approach
Measure T2I uncertainty in the text space: an LVLM captions the generated image, and the caption is compared with the original prompt using ROUGE or BERTScore; recall captures epistemic uncertainty and precision captures aleatoric uncertainty.
Key results
- PUNC with ROUGE recall achieves average AUROC of about 90.75% on the Texture OOD setting versus 70.5% for DDPM-OOD and 23.9% for 2XDM-style baselines in the averaged table (Table 3).
- On the Vague aleatoric setting, PUNC ROUGE precision reaches about 99.99-100% AUROC across SDXS, PixArt, SDv1.5 and SDXL, while image-space baselines stay near 40-57%.
- On corrupted prompts (Level 1), PUNC BERT precision achieves about 74% average AUROC versus roughly 47-50% for DDPM-OOD and LMD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PUNC with ROUGE-L recall instead of image-space uncertainty methods | Text-to-image uncertainty estimation · Microscopic | AUROC: 90.75% | Helped |
| Use PUNC with ROUGE-L recall instead of image-space uncertainty methods | Text-to-image uncertainty estimation · Remote Sensing | AUROC: 89.04% | Helped |
| Use PUNC with ROUGE-L recall instead of image-space uncertainty methods | Text-to-image uncertainty estimation · Texture | AUROC: 57.01% | Mixed |
| Use PUNC with ROUGE precision | Text-to-image uncertainty estimation · Vague | AUROC: 99.99% | Helped |
| Use PUNC with BERTScore precision | Text-to-image uncertainty estimation · Corrupt level 1 | AUROC: 74.00% | Helped |
| Use PUNC with BERTScore precision | Text-to-image uncertainty estimation · Corrupt level 2 | AUROC: 74.06% | Helped |
Surprising findings
- Image-space uncertainty baselines such as 2XDM often perform near or below chance (AUROC well under 50%) on several prompt-uncertainty settings.
Limitations
- Relies heavily on the interpretive abilities and possible biases of LVLMs.
- Analysis focuses on English-language prompts.
- Mixed results on the Texture OOD dataset, attributed to uncertainty about training data contents.
Experimental setup
Datasets: ImageNet prompts (Normal, from GPT-4 captions) Remote Sensing Texture Microscopic Vague Adversarial Corrupted
Benchmarks: Normal vs OOD datasets (Remote Sensing, Texture, Microscopic) Vague, Adversarial, Corrupted prompt datasets G7 politician recognition
Metrics: AUROC AUPR FPR95 accuracy
Baselines: DDPM-OOD LMD 2XDM
Builds on: DDPM-OOD LMD 2XDM
Closest papers by meaning
- FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges (NeurIPS 2025 · similarity 0.88)
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.88)
- Probabilistic Language-Image Pre-Training (ICLR 2025 · similarity 0.87)
- DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Models (NeurIPS 2026 · similarity 0.87)
- FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty (ICML 2026 · similarity 0.87)
- A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative Models (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.