Research map / Vision-language and multimodal / Multimodal LLMs
DiagramGPT-Llama3:Enabling Editable, High-Fidelity Diagram Generation with Vision Large Language Models
Yongyuan Chen, Minjie Hong, Boxi Wu, Xicheng Han
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes generating editable, high-fidelity diagrams (e.g., draw.io/XML-style) from text or raster images, introducing diagram restoration and style transfer tasks, using a vision LLM approach named DiagramGPT-Llama3.
Problem
Existing diagram generation from natural language often lacks editable output that supports user-friendly editing such as drag-and-drop.
Approach
A new task of generating editable, high-fidelity diagrams from either text or raster images, plus diagram restoration and style transfer tasks, addressed with a vision large language model built on Llama 3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add FABS to base model (ablation) | DiagramGPT base model · Diagram-mxGraph test | Overall score: 66.854 -> 66.988 | Improved |
| add CAAL to base model (ablation) | DiagramGPT base model · Diagram-mxGraph test | Overall score: 66.854 -> 66.966 | Improved |
| add both FABS and CAAL (ablation) | DiagramGPT base model · Diagram-mxGraph test | Overall score: 66.854 -> 67.188 | Improved |
| add CAAL to base model (ablation) | DiagramGPT base model · Diagram-mxGraph test | Captioning CIDEr: 4.54 -> 9.16 | Improved |
| add FABS alone to base model (ablation) | DiagramGPT base model · Diagram-mxGraph test | Captioning CIDEr: 4.54 -> 3.44 | Hurt |
| add CAAL to base model (ablation) | DiagramGPT base model · Diagram-mxGraph test, Task 3 | TaskScore: 52.190 -> 52.887 | Improved |
| add FABS to base model (ablation) | DiagramGPT base model · Diagram-mxGraph test, Task 1 | TaskScore: 95.780 -> 95.809 | No clear effect |
| add CAAL to base model (ablation) | DiagramGPT base model · Diagram-mxGraph test, Task 1 | TaskScore: 95.780 -> 95.094 | Hurt |
| add FABS and CAAL to base model (ablation) | DiagramGPT base model · Diagram-mxGraph test, Task 1 | TaskScore: 95.780 -> 95.298 | Hurt |
| fine-tune DiagramGPT instead of few-shot ChatGPT-4o-mini | diagram generation · Diagram-mxGraph test, Task 3 | TaskScore: 55.006 -> 52.739 | Hurt |
| use DiagramGPT instead of one-shot ChatGPT-4o-mini | diagram generation · human evaluation Task 1 accuracy | accuracy (0-100): 59.670 -> 73.977 | Improved |
Closest papers by meaning
- VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editing (ICML 2026 · similarity 0.89)
- From Words to Structured Visuals: A Benchmark and Framework for Text-to-Diagram Generation and Editing (CVPR 2025 · similarity 0.87)
- Text to Sketch Generation with Multi-Styles (NeurIPS 2025 · similarity 0.84)
- Factuality Matters: When Image Generation and Editing Meet Structured Visuals (ICLR 2026 · similarity 0.83)
- Primitive Vision: Improving Diagram Understanding in MLLMs (ICML 2025 · similarity 0.83)
- Scene Graph Disentanglement and Composition for Generalizable Complex Image Generation (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
