Research map / Image generation / Text-to-image diffusion
FlashEval: Towards Fast and Accurate Evaluation of Text-to-image Diffusion Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlashEval, an evolutionary-style iterative search that condenses text-to-image evaluation prompt sets (e.g., 50 prompts instead of 500) to rank diffusion models with high Kendall's Tau at ~10x speedup.
Problem
Full evaluation of text-to-image diffusion models on large prompt sets (COCO 40K, DiffusionDB) is too costly for iterative evaluation, while random subsampling yields poor ranking correlation at small sizes.
Approach
Search for representative prompt subsets directly optimizing ranking correlation (Kendall's Tau) with an iterative set- and prompt-level search, using frequency-based prompt selection among top-performing random subsets as the population update, rather than textual-feature coverage or per-prompt top-K selection.
Key results
- A searched 50-item COCO subset achieves Kendall's Tau comparable to a randomly sampled 500-item subset on unseen models (CLIP-Score test 0.851 vs 0.829), a claimed 10x evaluation speedup.
- On DiffusionDB, FlashEval's 5-item subset surpasses the average Kendall's Tau of random 100-item subsets on test models (0.789 vs 0.763 for random sampling at N'=100 in CLIP-Score).
- Random sampling of 10 items gives Kendall's Tau 0.142 on test models in ablation, versus 0.796 for FlashEval at N'=10 on COCO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FlashEval rather than B3-set to select a 50-prompt evaluation subset | Diffusion model ranking with CLIP-Score · COCO val2014, test split, random, model-variant, and scheduler sub-tasks | Kendall’s Tau: 0.750±0.014 -> 0.851±0.004; 0.680±0.021 -> 0.800±0.008; 0.721±0.013 -> 0.850±0.005 | Helped |
| Use FlashEval rather than B3-set to select a 500-prompt evaluation subset | Diffusion model ranking with CLIP-Score · COCO val2014, test split, random, model-variant, and scheduler sub-tasks | Kendall’s Tau: 0.875±0.007 -> 0.906±0.003; 0.836±0.008 -> 0.899±0.004; 0.863±0.008 -> 0.909±0.003 | Helped |
| Use FlashEval rather than B3-set to select a 5-prompt evaluation subset | Diffusion model ranking with CLIP-Score · DiffusionDB, test split, random, model-variant, and scheduler sub-tasks | Kendall’s Tau: 0.643±0.031 -> 0.789±0.009; 0.516±0.033 -> 0.691±0.011; 0.601±0.018 -> 0.748±0.014 | Helped |
| Use FlashEval rather than B3-set to select a 100-prompt evaluation subset | Diffusion model ranking with CLIP-Score · DiffusionDB, test split, random, model-variant, and scheduler sub-tasks | Kendall’s Tau: 0.826±0.010 -> 0.851±0.007; 0.792±0.018 -> 0.861±0.007; 0.836±0.014 -> 0.873±0.007 | Helped |
| Add frequency-based prompt selection (ablation) | FlashEval · COCO val2014, test split, N′=10 and N′=100 | Kendall’s Tau (CLIP-Score): 0.640 -> 0.796; 0.760 -> 0.867 | Helped |
| Add iterative filtering (ablation) | FlashEval · COCO val2014, test split, N′=10 and N′=100 | Kendall’s Tau (CLIP-Score): 0.667 -> 0.796; 0.812 -> 0.867 | Helped |
Surprising findings
- Textual-feature-based subset selection (category or distribution coverage) is inadequate for representative evaluation subsets, and insights from training-data coreset selection do not transfer.
- Prompt-wise top-K search does not improve with larger subset size, because combining individually high-KD prompts can yield low set-level KD.
Limitations
- Ground-truth rankings rely on the model zoo of 78 Stable Diffusion configurations, which is limited in architectural diversity.
- Requires running all candidate models on the full dataset to build the search, so one-time search cost remains.
- Generalization is evaluated on a limited set of unseen models and metrics.
Experimental setup
Datasets: COCO val2014 DiffusionDB
Benchmarks: COCO val2014 DiffusionDB
Metrics: Kendall's Tau top-K ranking top-K proportion
Baselines: random sampling B3-prompt (KD-based prompt-wise search) B3-set (KD-based set-wise random search)
Builds on: coreset selection evolutionary algorithms Kendall's Tau
Closest papers by meaning
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.90)
- CommonCanvas: Open Diffusion Models Trained on Creative-Commons Images (CVPR 2024 · similarity 0.89)
- On improved Conditioning Mechanisms and Pre-training Strategies for Diffusion Models (NeurIPS 2024 · similarity 0.89)
- InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (ICLR 2024 · similarity 0.89)
- On Discrete Prompt Optimization for Diffusion Models (ICML 2024 · similarity 0.89)
- UniFL: Improve Latent Diffusion Model via Unified Feedback Learning (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.