Research map / Image generation / Text-to-image diffusion
STEPS: Sequential Probability Tensor Estimation for Text-to-Image Hard Prompt Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STEPS, which casts discrete hard prompt search for text-to-image diffusion models as sequential probability tensor estimation using tensor-train nonnegative cores, with a memory-bounded resampling strategy.
Problem
Optimizing discrete text prompts for text-to-image diffusion models is hard because tokens are discrete, the search space is huge, and existing methods suffer from quantization error (embedding projection) or local optima (coordinate-wise greedy gradient search).
Approach
Models the distribution over prompt token configurations with nonnegative tensor-train cores, fitted by NLL on the minimal-loss sampled prompts, and grows then memory-bounds the vocabulary sets per position using gradient-selected candidates so the search space no longer grows with iteration count.
Key results
- On SD v2 with 15-token prompts, STEPS reaches 0.6720 on MS COCO vs 0.6485 for MMA and 0.6356 for PEZ.
- Transferred to SDXL v1.0 with 15-token prompts, STEPS scores 0.6245 on MS COCO and 0.6327 on Flickr30k, above MMA and PEZ.
- On I2P nudity prompts, STEPS attains 78.50% average ASR on SD v2 (black box) vs 72.71% for MMA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use STEPS instead of the strongest baseline for 15-token prompts | SD v2 prompt optimization · MS COCO | cosine similarity: 0.6356 -> 0.6720 | Helped |
| use STEPS instead of the strongest baseline for 15-token prompts | SD v2 prompt optimization · Flick30k | cosine similarity: 0.6288 -> 0.6775 | Helped |
| use STEPS instead of the strongest baseline for 15-token prompts | SD v2 prompt optimization · LAION | cosine similarity: 0.5583 -> 0.5868 | Helped |
| use STEPS instead of the strongest baseline for 15-token prompts | SD v2 prompt optimization · Celeb-A | cosine similarity: 0.5518 -> 0.5983 | Helped |
| use STEPS instead of the strongest baseline for 15-token prompts | SD v2 prompt optimization · textVQA | cosine similarity: 0.5770 -> 0.6053 | Helped |
| use STEPS instead of the strongest baseline for 15-token prompts | SD v2 prompt optimization · VizWiz | cosine similarity: 0.5378 -> 0.5722 | Helped |
| use STEPS instead of the strongest baseline for 15-token prompts | SDXL v1.0 cross-model prompt transfer · MS COCO | cosine similarity: 0.5641 -> 0.6245 | Helped |
| use STEPS instead of the strongest baseline for 15-token prompts | SDXL v1.0 cross-model prompt transfer · Flick30k | cosine similarity: 0.5580 -> 0.6327 | Helped |
| use STEPS instead of the strongest baseline for 15-token prompts | SDXL v1.0 cross-model prompt transfer · LAION | cosine similarity: 0.5185 -> 0.5630 | Helped |
| use STEPS instead of the strongest baseline for 15-token prompts | SDXL v1.0 cross-model prompt transfer · Celeb-A | cosine similarity: 0.3861 -> 0.4819 | Helped |
| use STEPS instead of the strongest baseline for 15-token prompts | SDXL v1.0 cross-model prompt transfer · textVQA | cosine similarity: 0.4963 -> 0.5454 | Helped |
| use STEPS instead of the strongest baseline for 15-token prompts | SDXL v1.0 cross-model prompt transfer · VizWiz | cosine similarity: 0.4305 -> 0.4867 | Helped |
Surprising findings
- Black-box ASR on SD v2 and SLD is sometimes higher than white-box ASR on SD v1.5, which the authors attribute to stronger generators producing more pronounced nudity.
Limitations
- Evaluation relies on CLIP-embedding cosine similarity rather than human or semantic judgments of images.
- Experiments use only 100 sampled images per dataset and 61 NSFW prompts, a small evaluation scale.
- Hardware and compute cost are not reported.
Experimental setup
Datasets: MS COCO Flickr30k LAION Celeb-A TextVQA VizWiz I2P
Benchmarks: MS COCO Flickr30k LAION Celeb-A TextVQA VizWiz I2P
Metrics: cosine similarity between generated image and target embeddi attack success rate (NudeNet, Q16, MHSC)
Baselines: PEZ MMA Genetic Original (SLD I2P prompts)
Builds on: PEZ AutoPrompt-style gradient token search (GCG-style one-hot gradients) tensor train decomposition TT-based probability estimation
Closest papers by meaning
- On Discrete Prompt Optimization for Diffusion Models (ICML 2024 · similarity 0.90)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.89)
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- Minority-Focused Text-to-Image Generation via Prompt Optimization (CVPR 2025 · similarity 0.88)
- A User-Friendly Framework for Generating Model-Preferred Prompts in Text-to-Image Synthesis (AAAI 2024 · similarity 0.88)
- Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.