Research map / Image generation / Text-to-image diffusion
PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PQPP, a human-annotated benchmark of 10,200 prompts/queries scored for both text-to-image generation (SDXL, GLIDE) and retrieval (CLIP, BLIP-2), and evaluates pre- and post-generation/retrieval difficulty predictors.
Problem
Predicting how difficult a text prompt is for text-to-image generation (and retrieval) has not been studied with human relevance judgments; prior work relied on automatic ground truth.
Approach
Jointly annotate the same 10,200 prompts/queries (DrawBench plus k-means-selected MS COCO captions) with human relevance judgments for both generated images (SDXL, GLIDE) and retrieved images (CLIP, BLIP-2), enabling cross-task and cross-model QPP evaluation with pre- and post-hoc predictors.
Key results
- Fine-tuned CLIP predictor reaches Pearson 0.649 for GLIDE HBPP generation performance, the highest correlation in the benchmark.
- Correlation between generation and retrieval difficulty is very low (e.g., HBPP vs. P@10 Pearson 0.135), suggesting the tasks are largely orthogonal.
- Fine-tuned BERT, which never sees images, is a competitive predictor, ranking second on generation and surpassing post-retrieval predictors in some cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use fine-tuned CLIP rather than the other tested predictors | Prompt performance prediction for GLIDE · GLIDE | Pearson correlation: 0.649‡ | Helped |
| Use fine-tuned CLIP rather than the other tested predictors | Prompt performance prediction for SDXL · SDXL | Pearson correlation: 0.380‡ | Helped |
| Use fine-tuned CLIP rather than the other tested predictors | Query performance prediction for CLIP retrieval · CLIP retrieval P@10 | Pearson correlation: 0.473‡ | Helped |
| Use fine-tuned BERT rather than the other tested predictors | Query performance prediction for CLIP retrieval · CLIP retrieval RR | Pearson correlation: 0.221‡ | Helped |
| Use fine-tuned BERT rather than the other tested predictors | Query performance prediction for BLIP-2 retrieval · BLIP-2 retrieval P@10 | Pearson correlation: 0.511‡ | Helped |
| Use correlation CNN rather than the other tested predictors | Query performance prediction for BLIP-2 retrieval · BLIP-2 retrieval RR | Pearson correlation: 0.206‡ | Helped |
Surprising findings
- Pre-retrieval fine-tuned BERT, which does not see generated or retrieved images, is competitive with post-generation/retrieval predictors.
- Predictors show large score drops when transferred across generative models, though retrieval predictors transfer more consistently.
Limitations
- Retrieval relevance labels are semi-automatic, seeded by Sentence-BERT similarity thresholds and limited to 2,000 images per query.
- Generation performance is measured only for two diffusion models (SDXL, GLIDE) and two retrieval models (CLIP, BLIP-2).
Experimental setup
Datasets: MS COCO DrawBench
Benchmarks: PQPP
Metrics: Pearson correlation Kendall correlation
Baselines: #synsets #words Fine-tuned BERT Fine-tuned CLIP Correlation CNN
Builds on: Long-CLIP Sun et al. correlation-based CNN QPP BERT
Closest papers by meaning
- TIGeR: Unifying Text-to-Image Generation and Retrieval with Large Multimodal Models (ICLR 2025 · similarity 0.88)
- GenIR: Generative Visual Feedback for Mental Image Retrieval (NeurIPS 2025 · similarity 0.87)
- Fix Before Search: Benchmarking Agentic Visual Query Pre-processing in Multimodal Retrieval-augmented Generation (ICML 2026 · similarity 0.86)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.86)
- Sentence-level Prompts Benefit Composed Image Retrieval (ICLR 2024 · similarity 0.86)
- Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.