Research map / Image generation / Text-to-image diffusion
GlyphDraw2: Automatic Generation of Complex Glyph Posters with Diffusion Models and Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GlyphDraw2, an SDXL-based bilingual poster generator with triple cross-attention (glyph and ControlNet cross-attention), an auxiliary alignment loss, and LLM (Baichuan2-13B) fine-tuned to predict glyph bounding boxes from captions, plus new poster and font datasets.
Problem
Accurately rendering paragraph-level bilingual (Chinese and English) text in diffusion-generated images and automatically producing poster layouts from user prompts without manual bounding-box input, while keeping background richness.
Approach
Adds two extra cross-attention layers to the SDXL decoder (one attending to InternViT glyph features, one to ControlNet features), trains an auxiliary alignment loss on a duplicated decoder to preserve background semantics, and uses a fine-tuned LLM to generate glyph text and bboxes automatically.
Key results
- On AnyText-Benchmark English, GlyphDraw2 reaches Acc 0.8627 vs 0.7108 for AnyText-v1.1 and 0.6435 for UDiffText.
- On Complex-Benchmark English with LLM-predicted layouts, GlyphDraw2 reaches Acc 0.4158 vs 0.1856 for LLMs+ControlNet and 0.0999 for TextDiffuser-2.
- On Poster-Benchmark Chinese, GlyphDraw2 reaches Acc 0.8263 and NED 0.9585, close to Glyph-ByT5 (0.8248 Acc) while Glyph-ByT5 has higher ClipScore.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GlyphDraw2 instead of GlyphDraw1.1 | GlyphDraw · AnyText-Benchmark Chinese | Acc: 0.7892 -> 0.8266 | Helped |
| Use GlyphDraw2 instead of ControlNet w/ canny | text rendering model · AnyText-Benchmark English | Acc: 0.7954 -> 0.8627 | Helped |
| Use GlyphDraw2 instead of UDiffText | text rendering model · ICDAR13 English | Acc: 0.5840 -> 0.6901 | Helped |
| Use GlyphDraw2 instead of TextDiffuser | text rendering model · MARIO-Eval English | Acc: 0.5609 -> 0.7672 | Helped |
| Use GlyphDraw2 instead of GlyphDraw1.1 | automatic poster generation · Complex-Benchmark Chinese | Acc: 0.6215 -> 0.6691 | Helped |
| Use GlyphDraw2 instead of GlyphDraw1.1 | automatic poster generation · Complex-Benchmark English | Acc: 0.2264 -> 0.4158 | Helped |
| Use GlyphDraw2 instead of Glyph-ByT5 | poster generation · Poster-Benchmark English | Acc: 0.7341 -> 0.7590 | Helped |
| Add CAG (ablation) | GlyphDraw2 · Poster-Benchmark Chinese | Acc: 0.7841 -> 0.8161 | Helped |
| Add CAC (ablation) | GlyphDraw2 · Poster-Benchmark Chinese | Acc: 0.7985 -> 0.8161 | Helped |
| Add TCA (ablation) | GlyphDraw2 · Poster-Benchmark Chinese | Acc: 0.7802 -> 0.8058 | Helped |
| Add AAL (ablation) | GlyphDraw2 · Poster-Benchmark Chinese | ClipScore: 0.3884 -> 0.4099 | Helped |
| Add FTE (ablation) | GlyphDraw2 · Poster-Benchmark Chinese | Acc: 0.7965 -> 0.8161 | Helped |
Surprising findings
- FLUX.1 and SD3 achieve better ClipScore/HPSv2 than GlyphDraw2 on English benchmarks despite weaker or comparable text accuracy on some sets.
- Glyph-ByT5 achieves higher ClipScore and HPSv2 than GlyphDraw2 on AnyText-Benchmark, though authors note its renderings sometimes fall outside the bbox.
Limitations
- LLM bbox prediction is inaccurate for complex scenarios such as unquoted paragraph text.
- Prioritizing glyph accuracy weakens background richness and visual appeal.
- Accuracy remains low for tiny glyphs and paragraph-level text; Canny edge noise causes shadows and blurred small text.
Experimental setup
Datasets: AnyText-Benchmark ICDAR13 MARIO-Eval LAION-Glyph MARIO-10M AnyWord-3M LAION Wukong
Benchmarks: AnyText-Benchmark ICDAR13 MARIO-Eval Complex-Benchmark Poster-Benchmark
Metrics: Accuracy Normalized Edit Distance ClipScore HPSv2
Baselines: SD3 Kolors FLUX.1-schnell ControlNet ControlNet w/ canny TextDiffuser AnyText-v1.1 UDiffText Glyph-ByT5 GlyphDraw1.1 TextDiffuser-2
Builds on: SDXL ControlNet IP-Adapter GlyphDraw AnyText PEA-Diffusion
Closest papers by meaning
- GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editing (CVPR 2025 · similarity 0.88)
- POSTA: A Go-to Framework for Customized Artistic Poster Generation (CVPR 2025 · similarity 0.88)
- GlyphAnchor: Enhancing Visual Text Rendering via Position-Anchored Glyph Priors (NeurIPS 2026 · similarity 0.86)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.86)
- Paper2Poster: Towards Multimodal Poster Automation from Scientific Papers (NeurIPS 2025 · similarity 0.86)
- VecGlypher: Unified Vector Glyph Generation with Language Models (CVPR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.