Research map / Image generation / Text-to-image diffusion
Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertainty
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes proactive text-to-image agents that ask clarification questions and maintain an editable probabilistic belief graph, evaluated with a simulated-user self-play pipeline and the new DesignBench dataset.
Problem
Text-to-image prompts are underspecified, so models make implicit assumptions that misalign with user intent, forcing repeated manual prompt refinement.
Approach
Agents maintain a graph-based belief state (entities, attributes, relations with probabilities and importance scores), built by an LLM, and use it to choose and phrase clarification questions over multiple turns before calling a frozen T2I model.
Key results
- On DesignBench, Ag3 raises VQAScore T2I from 0.353 (single-turn T2I) to 0.955 after 15 turns.
- Human raters preferred agent-generated images over single-turn T2I in 80%+ of cases for content and aesthetics.
- In a 143-person survey, 91% rated clarifications as could/likely/very likely helpful.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace single-turn prompting with multi-turn clarification (Ag3) | Imagen 3 T2I generation · Coco-Captions | VQAScore: 0.298$\pm.5$ -> 0.775$\pm.4$ | Helped |
| Replace single-turn prompting with multi-turn clarification (Ag3) | Imagen 3 T2I generation · ImageInWords | VQAScore: 0.371$\pm.5$ -> 0.662$\pm.5$ | Helped |
| Replace single-turn prompting with multi-turn clarification (Ag3) | Imagen 3 T2I generation · DesignBench | VQAScore: 0.353$\pm.5$ -> 0.955$\pm.2$ | Helped |
| Use principle-guided LLM questions instead of belief-conditioned questions (ablation) | Ag2 · Coco-Captions | VQAScore: 0.663$\pm.5$ -> 0.775$\pm.4$ | Helped |
| Use principle-guided LLM questions instead of belief-conditioned questions (ablation) | Ag2 · ImageInWords | VQAScore: 0.570$\pm.5$ -> 0.662$\pm.5$ | Helped |
| Use principle-guided LLM questions instead of belief-conditioned questions (ablation) | Ag2 · DesignBench | VQAScore: 0.855$\pm.3$ -> 0.955$\pm.2$ | Helped |
| Use principle-guided LLM questions instead of heuristic question scoring (ablation) | Ag1 · Coco-Captions | VQAScore: 0.583$\pm.5$ -> 0.775$\pm.4$ | Helped |
| Use principle-guided LLM questions instead of heuristic question scoring (ablation) | Ag1 · ImageInWords | VQAScore: 0.5058$\pm.5$ -> 0.662$\pm.5$ | Helped |
| Use principle-guided LLM questions instead of heuristic question scoring (ablation) | Ag1 · DesignBench | VQAScore: 0.685$\pm.5$ -> 0.955$\pm.2$ | Helped |
| Run additional interaction turns | Multi-turn agents · ImageInWords | T2I VQAScore | Mixed |
Surprising findings
- The simpler principle-prompted agent (Ag3, no explicit belief graph) outperformed the belief-prompted agent (Ag2) on nearly all metrics.
- Text-text and text-image similarity plateau or decrease after about 10 turns while image-image similarity keeps increasing.
Limitations
- NLL approximation assumes independence of entities, attributes and relations and ignores name similarity.
- Belief graph quality depends on LLM parsing; importance-score errors degrade question quality (Ag2).
- Evaluation relies on simulated users rather than real human conversations for automatic metrics.
Experimental setup
Datasets: COCO-Captions ImageInWords
Benchmarks: COCO-Captions ImageInWords DesignBench
Metrics: EmbedSim T2T DSG T2T DINO I2I VQAScore T2I NLL human rating
Baselines: single-turn T2I (Imagen 3) Ag1 Ag2
Builds on: VQAScore DSG DINOv2 ImageInWords
Closest papers by meaning
- Preference Adaptive and Sequential Text-to-Image Generation (ICML 2025 · similarity 0.87)
- Refine and Align: Confidence Calibration Through Multi-Agent Interaction in VQA (AAAI 2026 · similarity 0.87)
- Gen-Searcher: Reinforcing Agentic Search for Image Generation (NeurIPS 2026 · similarity 0.85)
- Rich Human Feedback for Text-to-Image Generation (CVPR 2024 · similarity 0.85)
- Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following (CVPR 2024 · similarity 0.85)
- What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generation (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.