Research map / Image generation / Text-to-image diffusion
What's Producible May Not Be Reachable: Measuring the Steerability of Generative Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a decomposition separating generative model steerability from producibility, a goal-reproduction benchmark for steering text-to-image models and LLMs with human users, and an image-based steering mechanism that doubles improvement over text steering.
Problem
Existing generative model evaluations measure producibility (quality and breadth of outputs) but not steerability (whether users with specific goals can steer the model to outputs satisfying them), and human-AI interaction benchmarks blend the two.
Approach
Decompose the reward gap into a producibility gap and a steerability gap; measure steerability by sampling a goal output from the model and asking users to reproduce it, judged by separate human raters; then propose image steering (suggesting image variations chosen by a learned RL proposal distribution) in place of text-only prompt rewriting.
Key results
- Image steering with learned proposals reaches 74.2% improvement rate and 0.053 average DreamSim change on PixelProse, versus 54.7% and 0.025 for text steering.
- Annotators rated attempted text-to-image reproductions unsatisfactory 60% of the time; after five attempts the final image was closer to the goal only 62% of the time versus a 50% base rate.
- A blind LLM rewriting the first prompt achieves 52% of human improvement with equal attempts and 87% with 20 attempts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace prompt rewrites with learned image proposals | Stable Diffusion 1.4 image steering · PixelProse | DreamSim average change: 0.025 (0.010) -> 0.053 (0.008) | Helped |
| Replace prompt rewrites with learned image proposals | Stable Diffusion 1.4 image steering · Tiles | DreamSim average change: 0.029 (0.015) -> 0.072 (0.007) | Helped |
| Use learned rather than random image proposals (ablation) | Stable Diffusion 1.4 image steering · PixelProse | DreamSim average change: 0.040 (0.007) -> 0.053 (0.008) | Helped |
| Use learned rather than random image proposals (ablation) | Stable Diffusion 1.4 image steering · Tiles | DreamSim average change: 0.053 (0.005) -> 0.072 (0.007) | Helped |
| Allow users to choose random seeds (ablation) | Text-to-image steering · Text-to-image steering benchmark | DreamSim similarity: 0.672 (0.003) -> 0.674 (0.008) | No clear effect |
| Allow users to specify classifier-free guidance settings (ablation) | Stable Diffusion 3.5 Medium text steering · Stable Diffusion 3.5 Medium | DreamSim similarity: 0.670 (0.008) -> 0.679 (0.007) | No clear effect |
| Use professional prompt engineers rather than regular users (ablation) | Stable Diffusion 3.5 Large Turbo text steering · Stable Diffusion 3.5 Large Turbo | DreamSim similarity at attempt 5: 0.670 (0.016) -> 0.741 (0.025) | Helped |
| Constrain the available random seeds (ablation) | Stable Diffusion 3.5 Large Turbo · Steerability and producibility evaluation | DreamSim similarity | Mixed |
| Use text steering rather than image steering | Stable Diffusion 1.4 · PixelProse and Tiles | DreamSim average change: PixelProse: 0.053 (0.008) vs 0.025 (0.010); Tiles: 0.072 (0.007) vs 0.029 (0.015) | Hurt |
Surprising findings
- Professional prompt engineers only slightly outperform non-experts (about 10% higher final similarity) and show no reliable improvement across attempts.
- Blind LLM prompt variations without access to the goal image capture a large share of human steering improvement, suggesting apparent learning is partly lottery effects.
Limitations
- Only a basic implementation of image steering and a single steering proposal learner were tested, on a single 860M-parameter model (Stable Diffusion 1.4).
- Steerability is measured over single interaction sessions; longer time horizons are not studied.
- Metrics rely on human annotator judgments and similarity measures that may be subject to variability.
Experimental setup
Datasets: PixelProse Kaggle News Category dataset Tiles prompts generated by Claude-3.5-Sonnet
Benchmarks: steerability benchmark (PixelProse goal images) steerability benchmark (Tiles goal images) LLM headline steerability benchmark
Metrics: satisfaction rate image similarity rating improvement rate prompt-output misalignment DreamSim CLIP similarity
Baselines: text steering random image proposals DALL-E 2 DALL-E 3 Stable Diffusion 3.5 Large Turbo
Builds on: ideal point reward function DreamSim PixelProse
Closest papers by meaning
- Concept Reachability in Diffusion Models: Beyond Dataset Constraints (ICML 2025 · similarity 0.87)
- IDEA-Bench: How Far are Generative Models from Professional Designing? (CVPR 2025 · similarity 0.87)
- Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences (NeurIPS 2024 · similarity 0.86)
- Analysing the Generalisation and Reliability of Steering Vectors (NeurIPS 2024 · similarity 0.86)
- ArtWhisperer: A Dataset for Characterizing Human-AI Interactions in Artistic Creations (ICML 2024 · similarity 0.86)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.