Research map / Image generation / Text-to-image diffusion
ChatGen: Automatic Text-to-Image Generation From FreeStyle Chatting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Automatic T2I, a benchmark (ChatGenBench) and a three-stage training method (ChatGen-Evo) where an MLLM rewrites freestyle chat into prompts, selects a T2I model via special ModelTokens, and configures arguments via in-context learning.
Problem
Non-expert users face trial-and-error when generating images with text-to-image models because they must craft prompts, pick among many community models, and configure arguments; existing methods automate only one step and do not accept freestyle chat input.
Approach
Decompose automatic T2I into three stages (prompt writing via SFT, model selection via learned ModelTokens appended to the MLLM vocabulary, argument configuration via in-context learning with model demonstrations) and build a step-wise benchmark from Civitai demonstrations with LLM role-play to create freestyle inputs.
Key results
- ChatGen-Evo (2B) reaches a Unified Metric of 65.9 on the supervised setting versus 60.7 for ChatGen-Base (8B), with selection accuracy 0.328 vs 0.264.
- Argument configuration accuracy rises from 53.7% to 87.1% when ground-truth prompt and model are given in the ablation with ChatGen-Evo.
- In a human pairwise study, ChatGen-Evo won over ChatGen-Base in the few-shot setting with a 71% win rate on relevance and 53% on quality (as reported in Figure 4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-stage evolution instead of direct SFT prediction | ChatGen-Base with InternVL2-2B · ChatGenBench Supervised | Unified Metric: 60.7 -> 65.9 | Helped |
| use multi-stage evolution instead of direct SFT prediction | ChatGen-Base with InternVL2-2B · ChatGenBench Few-shot | Unified Metric: 52.5 -> 59.2 | Helped |
| provide the human-validated prompt, model, and argument (ablation) | ChatGen-Evo · ChatGenBench Supervised | Config Acc: 0.537 -> 0.871 | Helped |
| provide the human-validated prompt and argument for model selection (ablation) | ChatGen-Evo · ChatGenBench Supervised | Selection Acc: 32.8% -> 55.3% | Helped |
| use the predicted prompt rather than the human-validated prompt (ablation) | ChatGen-Evo · ChatGenBench Supervised | Unified Score: 70.3 -> 66.8 | Hurt |
| use historical dialogue input rather than single input | ChatGen-Evo · ChatGenBench History | Unified Score: 68.1 -> 60.1 | Hurt |
| use multimodal input rather than single input | ChatGen-Evo · ChatGenBench M-Modal | Unified Score: 68.1 -> 69.3 | Helped |
Surprising findings
- Model selection accuracy depends heavily on prompt quality: with human-validated prompts, ChatGen-Evo's selection accuracy rises from 32.8% to 55.3%.
- ChatGen-Evo at 2B matches ChatGen-Base at 8B in performance, yet is slower at inference (1.9s vs 2.3s per sample is faster, but training takes 100h vs 240h).
Limitations
- Step-wise ground truth is relative (human-validated demonstrations) rather than absolute, given the infinite search space
- Few-shot setting shows marked declines in model selection and argument configuration
- Historical multi-turn inputs yield the lowest performance across metrics
Experimental setup
Datasets: Civitai demonstrations ChatGenBench TrainSet
Benchmarks: ChatGenBench
Metrics: BERTScore selection accuracy argument accuracy FID CLIP Score HPS v2 ImageReward Unified Metric win rate (human study)
Baselines: ChatGen-Base Baseline without fine-tuning (MLLM ICL + default Stable Diffusion) BeautifulPrompt DiffChat DiffusionGPT DABench
Builds on: InternVL2 BeautifulPrompt DiffAgent Stable Diffusion
Closest papers by meaning
- CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation (ICML 2025 · similarity 0.90)
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.88)
- Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following (CVPR 2024 · similarity 0.88)
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.87)
- Revisiting text-to-image evaluation with Gecko: on metrics, prompts, and human rating (ICLR 2025 · similarity 0.87)
- Rich Human Feedback for Text-to-Image Generation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.