Research map / Image generation / Text-to-image diffusion
From Words to Structured Visuals: A Benchmark and Framework for Text-to-Diagram Generation and Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces DiagramGenBenchmark (eight diagram types) and DiagramAgent, a multi-agent framework (Plan, Code, Check, Diagram-to-Code agents) for generating and editing structured diagrams from text via LaTeX/DOT code.
Problem
Text-to-image models produce visually appealing but logically incoherent outputs, and text-to-code approaches struggle with complex, editable diagrams; there is no comprehensive dataset for generating and editing diverse structured diagrams.
Approach
A multi-agent pipeline in which a planner expands queries, a fine-tuned code model writes TikZ/DOT code, a Check Agent compiles and verifies it with GPT-4o, and a Diagram-to-Code agent enables editing by recovering code from an existing diagram.
Key results
- DiagramAgent Code Agent reaches Pass@1 58.15, ROUGE-L 51.97, CodeBLEU 86.83 on diagram generation, outperforming GPT-4o (49.81 Pass@1).
- On diagram coding, DiagramAgent achieves Pass@1 68.89 versus 64.07 for GPT-4o and 50.74 for Deepseek-vl.
- On diagram editing, DiagramAgent reaches Pass@1 98.00 and ROUGE-L 98.41, versus 95.00 Pass@1 for DeepSeek V2.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiagramAgent instead of the strongest listed baseline | Diagram generation with Qwen2.5-Coder-7B · Diagram generation | Pass@1: 55.56 -> 58.15 | Helped |
| Use DiagramAgent instead of the strongest listed baseline | Diagram coding with Qwen2-VL-7B · Diagram coding | Pass@1: 64.07 -> 68.89 | Helped |
| Use DiagramAgent instead of the strongest listed baseline | Diagram editing with Qwen2.5-Coder-7B · Diagram editing | Pass@1: 95.00 -> 98.00 | Helped |
| Add GPT-4o verification (ablation) | DiagramAgent diagram generation · Diagram generation | chrF: -0.68 | Helped |
| Add compiler debugging (ablation) | DiagramAgent diagram generation · Diagram generation | chrF: -1.75 | Helped |
| Add GPT-4o verification (ablation) | DiagramAgent diagram coding · Diagram coding | Pass@1: -6.30 | Helped |
| Add compiler debugging (ablation) | DiagramAgent diagram coding · Diagram coding | Pass@1: -15.56 | Helped |
| Add GPT-4o verification (ablation) | DiagramAgent diagram editing · Diagram editing | Pass@1: -1.00 | Helped |
| Add compiler debugging (ablation) | DiagramAgent diagram editing · Diagram editing | Pass@1: -2.50 | Helped |
Surprising findings
- Removing the GPT-4o verification module slightly improves PSNR in some generation settings and the paper attributes this to simpler representations.
- Diagram editing scored higher than diagram generation in human evaluation, which the authors attribute to editing starting from existing code.
Limitations
- Handling of complex structures with intricate spatial relationships remains limited.
- Diagram generation from scratch remains less accurate than editing and often requires iterations.
- Evaluation relies on a single-benchmark comparison and human raters with limited sample sizes.
Experimental setup
Datasets: VGQA datikz datikz-v2 GitHub and Overleaf diagram code repositories
Benchmarks: DiagramGenBenchmark
Metrics: Pass@1 ROUGE-L CodeBLEU Edit Distance chrF RUBY CLIP-FID LPIPS PSNR MS-SSIM
Baselines: Qwen2.5-Coder DeepSeek-Coder-Instruct Code-Llama WizardCoder GPT-4o DeepSeek V2.5 GLM-4-plus Gemini Qwen2-VL Deepseek-vl
Builds on: Qwen2.5-Coder Qwen2-VL GPT-4o
Closest papers by meaning
- VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editing (ICML 2026 · similarity 0.89)
- Factuality Matters: When Image Generation and Editing Meet Structured Visuals (ICLR 2026 · similarity 0.87)
- Text-to-Image Generation for Abstract Concepts (AAAI 2024 · similarity 0.86)
- ChartGalaxy: A Dataset for Infographic Chart Understanding and Generation (ICLR 2026 · similarity 0.86)
- AutoPresent: Designing Structured Visuals from Scratch (CVPR 2025 · similarity 0.86)
- ChartE$^{3}$: A Comprehensive Benchmark for End-to-End Chart Editing (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.