Research map / Image generation / Text-to-image diffusion
ArtWhisperer: A Dataset for Characterizing Human-AI Interactions in Artistic Creations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ArtWhisperer, an online prompt-guessing game that collected over 50,000 human-text-to-image interactions, and uses it to analyze prompt diversity, define a Markov-chain steerability metric, and evaluate GPT-4 and Gemini as prompters.
Problem
Existing human-AI interaction datasets for text-to-image models are single-shot and lack a fixed user goal, so they cannot reveal how people iteratively steer a model toward a target image or how steerable a model is.
Approach
Players are given a target image and iteratively write prompts to reproduce it using a fixed Stable Diffusion setup with a CLIP-based similarity score as feedback; the resulting sequential trajectories are modeled as a binned score Markov chain whose expected stopping time serves as a steerability metric.
Key results
- Over 50,000 interactions (51,026) from 2,250 players across 191 target images were recorded; the average score rose from 51.9 to 70.3 in the prompt-diversity analysis.
- Adjacent prompt changes improved score 40-60% of the time, with an average rate of 48.6%.
- GPT-4 improved steadily with feedback over about 15 rounds, while Gemini did not improve with feedback; the CLIP-based score correlated 0.579 with human ratings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Iterate on prompts using generated-image and score feedback | Users prompting Stable Diffusion v2.1 · ArtWhisperer | score: 56.7 -> 73.0 | Helped |
| Provide image and score feedback between prompt attempts | GPT-4 · ArtWhisperer target images | score | Helped |
| Provide image and score feedback between prompt attempts | Gemini · ArtWhisperer target images | score | No clear effect |
| Use BERT or GLOVE instead of CLIP text embeddings | Prompt-diversity analysis · ArtWhisperer | prompt-diversity findings | No clear effect |
| Assess steerability using human ratings instead of the automated score | Steerability analysis · ArtWhisperer-Validation image groups | steerability | No clear effect |
| Use SDv2.1 instead of SDv1.5 | Stable Diffusion · Nature, sci-fi or space, and real images | steerability | Helped |
| Use SDv2.1 instead of SDv1.5 | Stable Diffusion · Most other image categories | steerability | No clear effect |
| Have subject-matter knowledge of the target image | Human users steering Stable Diffusion · Famous people and landmarks | steerability | Helped |
| Increase output variance across seeds for a fixed prompt | Stable Diffusion · Target images | expected stopping time | Hurt |
| Improve prompts through small iterative updates | Human users · ArtWhisperer | score increase rate between adjacent prompts: 48.6% | Helped |
Surprising findings
- Prompt diversity does not decrease as users find higher-scoring prompts, even though image diversity does decrease.
- Gemini did not improve with feedback, which the authors attribute to it not being trained for multi-turn conversation.
Limitations
- Only 191 unique target images, limiting image diversity.
- The CLIP-based scoring function correlates only moderately (0.579) with human ratings.
- Players were anonymous and sessions not tracked across days; crowd-worker validation used a different setting.
Experimental setup
Datasets: ArtWhisperer ArtWhisperer-Validation Stable Diffusion Prompts dataset Wikipedia images LAION5B
Benchmarks: ArtWhisperer ArtWhisperer-Validation
Metrics: ArtWhisperer score expected stopping time (steerability) Pearson correlation prompt embedding distance score trajectory
Baselines: GPT-4 Gemini human users
Builds on: Stable Diffusion CLIP Promptist IA3
Closest papers by meaning
- Social Reward: Evaluating and Enhancing Generative AI through Million-User Feedback from an Online Creative Community (ICLR 2024 · similarity 0.88)
- Sketchtopia: A Dataset and Foundational Agents for Benchmarking Asynchronous Multimodal Communication with Iconic Feedback (CVPR 2025 · similarity 0.86)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.86)
- Preference Adaptive and Sequential Text-to-Image Generation (ICML 2025 · similarity 0.86)
- What's Producible May Not Be Reachable: Measuring the Steerability of Generative Models (NeurIPS 2025 · similarity 0.86)
- Multimodal Large Language Models Make Text-to-Image Generative Models Align Better (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.