Research map / Image generation / Editing and style transfer
DialogDraw: Image Generation and Editing System Based on Multi-Turn Dialogue
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents DialogDraw, a multi-turn dialogue system that routes user requests to ~20 open-source diffusion drawing pipelines via an intent-recognition and parameter-extraction model (Qwen-VL trained with SFT then DPO), plus a new DrawnConvos dataset and multi-turn metrics.
Problem
Existing LLM-based drawing workflows mostly rewrite prompts and rely on single models, so they fail to identify user intent across multi-turn dialogues for generating and editing images with diverse pipelines.
Approach
An intent recognition and parameter extraction model (IRPEM), built on Qwen-VL and trained with SFT then DPO on a dialogue dataset, outputs pipeline names and parameters that invoke existing diffusion workflows, so generation and editing happen inside those pipelines rather than in the LLM.
Key results
- DialogDraw achieves Instruct Edit Coherence 0.8196 and Multi-turn CLIP Similarity 0.7835, the best among compared models, with Multi-turn VQA Score 0.8491 second only to GPT-4 (0.8523).
- IRPEM(RLHF) reaches Intent Switch Accuracy 0.924 versus 0.702 for zero-shot Qwen-VL.
- IRPEM(RLHF) with pipelines beats DPO-only and SFT-only variants by 2.12% and 4.30% in IEC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DPO after SFT (ablation) | IRPEM on DrawnConvos(TEST) · DrawnConvos(TEST) | Intent Switch Accuracy: 0.902 -> 0.924 | Helped |
| add DPO after SFT (ablation) | IRPEM on DrawnConvos(TEST) · DrawnConvos(TEST) | Parameter Accuracy: 0.870 -> 0.892 | Helped |
| add DPO after SFT (ablation) | IRPEM on DrawnConvos(TEST) · DrawnConvos(TEST) | Structural Conformity: 0.932 -> 0.953 | Helped |
| add SFT before DPO (ablation) | IRPEM on DrawnConvos(TEST) · DrawnConvos(TEST) | Intent Switch Accuracy: 0.874 -> 0.924 | Helped |
| add SFT before DPO (ablation) | IRPEM on DrawnConvos(TEST) · DrawnConvos(TEST) | Parameter Accuracy: 0.857 -> 0.892 | Helped |
| add SFT before DPO (ablation) | IRPEM on DrawnConvos(TEST) · DrawnConvos(TEST) | Structural Conformity: 0.865 -> 0.953 | Helped |
| train with SFT and DPO rather than zero-shot (ablation) | Qwen-VL with drawing pipelines · DrawnConvos(TEST) | Instruct Edit Coherence: 0.7480 -> 0.8196 | Helped |
| train with SFT and DPO rather than SFT only (ablation) | Qwen-VL with drawing pipelines · DrawnConvos(TEST) | Instruct Edit Coherence: 0.8026 -> 0.8196 | Helped |
| train with SFT and DPO rather than DPO only (ablation) | Qwen-VL with drawing pipelines · DrawnConvos(TEST) | Instruct Edit Coherence: 0.7858 -> 0.8196 | Helped |
| add SFT and DPO training to zero-shot Qwen-VL (ablation) | Qwen-VL intent and parameter recognition · DrawnConvos(TEST) | Intent Switch Accuracy: 0.702 -> 0.924 | Helped |
| add SFT and DPO training to zero-shot Qwen-VL (ablation) | Qwen-VL intent and parameter recognition · DrawnConvos(TEST) | Parameter Accuracy: 0.754 -> 0.892 | Helped |
| add SFT and DPO training to zero-shot Qwen-VL (ablation) | Qwen-VL intent and parameter recognition · DrawnConvos(TEST) | Structural Conformity: 0.738 -> 0.953 | Helped |
Surprising findings
- Standalone SFT produced outputs with repetitions, garbled text, or negative sentiment, which DPO suppressed.
Limitations
- Integrates only about 20 pipelines, limiting drawing coverage.
- Limited progress on VQA and text dialogue capabilities.
- Human evaluation involved only 50 users and 100 dialogues.
Experimental setup
Datasets: DrawnConvos
Benchmarks: DrawnConvos(TEST)
Metrics: Multi-turn VQA Score Multi-turn CLIP Similarity Instruct Edit Coherence Human Score Intent Switch Accuracy Parameter Accuracy Structural Conformity
Baselines: SEED-LLaMA-8B SEED-LLaMA-14B GPT-4 DialogGen Qwen-VL-zero-shot
Builds on: DPO LoRA Qwen-VL VQAScore InstructPix2Pix
Closest papers by meaning
- SketchAgent: Language-Driven Sequential Sketch Generation (CVPR 2025 · similarity 0.89)
- SemanticDraw: Towards Real-Time Interactive Content Creation from Image Diffusion Models (CVPR 2025 · similarity 0.86)
- ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer (ICLR 2025 · similarity 0.85)
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.85)
- OmniGen: Unified Image Generation (CVPR 2025 · similarity 0.84)
- Janus-Pro-R1: Advancing Collaborative Visual Comprehension and Generation via Reinforcement Learning (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.