Research map / Image generation / Text-to-image diffusion
Layout-Agnostic Scene Text Image Synthesis with Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SceneTextGen, a layout-free latent diffusion model for scene text image generation that injects character-level features via cross-attention and trains with word-level OCR and character segmentation losses.
Problem
Diffusion-based scene text generation relies on predefined intermediate text layouts, which limits font style diversity and text placement, and generated text is often illegible or inaccurate.
Approach
Replace the predefined layout stage with a character-level encoder whose features are injected via hierarchical (sequential) cross-attention in the U-Net, plus an OCR word-spotting loss and a character segmentation loss applied to the predicted clean latent so the model learns text layout implicitly.
Key results
- SceneTextGen-7M reaches AP 0.5274 and accuracy 0.3088 on MARIO-7M-Eval, versus 0.5102 AP for ControlNet and 0.6135 AP for TextDiffuser-10M (which uses 10M data and layouts).
- On OpenLibrary-Eval, SceneTextGen-7M gets AP 0.4136, above TextDiffuser-7M (0.4257 is higher, so not all metrics favor it), while on TMDB-Eval it gets AP 0.3813 vs 0.3198 for TextDiffuser-7M.
- SceneTextGen-7M achieves the highest CLIPScore among compared models (0.3455 vs 0.3436 for TextDiffuser-10M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SceneTextGen-7M instead of the strongest listed baseline for AP | SceneTextGen-7M · MARIO-7M-Eval | AP: 0.6135 -> 0.5274 | Hurt |
| Use SceneTextGen-7M instead of ControlNet | SceneTextGen-7M · MARIO-TMDB-Eval | AR: 0.4667 -> 0.4716 | Helped |
| Use SceneTextGen-7M instead of ControlNet | SceneTextGen-7M · MARIO-OpenLibrary-Eval | AR: 0.4273 -> 0.4519 | Helped |
| Use SceneTextGen-7M instead of GlyphControl-10M | SceneTextGen-7M · Overall image-text alignment | CLIP Score: 0.3450 -> 0.3455 | Helped |
| Use both word-level and character-level losses rather than word-level loss only (ablation) | SceneTextGen-7M · MARIO-7M-Eval | AP: 0.5082 -> 0.5274 | Helped |
| Use both word-level and character-level losses rather than character-level loss only (ablation) | SceneTextGen-7M · MARIO-7M-Eval | AP: 0.5083 -> 0.5274 | Helped |
| Use both word-level and character-level losses rather than word-level loss only (ablation) | SceneTextGen-7M · MARIO-7M-Eval | AC: 0.3109 -> 0.3088 | Hurt |
| Use both word-level and character-level losses rather than character-level loss only (ablation) | SceneTextGen-7M · MARIO-7M-Eval | AC: 0.3035 -> 0.3088 | Helped |
| Set word and character loss weights to 1 and 2 rather than 0.01 and 0.02 (ablation) | SceneTextGen-7M · MARIO-7M-Eval | AP: 0.5274 -> 0.4107 | Hurt |
| Set word and character loss weights to 0.1 and 0.2 rather than 0.01 and 0.02 (ablation) | SceneTextGen-7M · MARIO-7M-Eval | AP: 0.5274 -> 0.4587 | Hurt |
| Set word and character loss weights to 1 and 2 rather than 0.01 and 0.02 (ablation) | SceneTextGen-7M · MARIO-7M-Eval | AC: 0.3088 -> 0.2782 | Hurt |
| Set word and character loss weights to 0.1 and 0.2 rather than 0.01 and 0.02 (ablation) | SceneTextGen-7M · MARIO-7M-Eval | AC: 0.3088 -> 0.2895 | Hurt |
Surprising findings
- The word-level OCR loss alone encouraged repetitive words, which the character-level segmentation loss was added to fix.
- Without predefined layouts, the model still outperforms layout-based baselines on several metrics, though TextDiffuser-10M has higher AP on MARIO-7M-Eval.
Limitations
- Struggles with complex scene elements combined with text (e.g., specific sticker color on a car).
- Text accuracy and coherence degrade for long text prompts.
- Relies on noisy OCR detection and recognition to create training bounding boxes and transcriptions.
Experimental setup
Datasets: MARIO-7M MARIO-TMDB MARIO-OpenLibrary LAION-Glyph LAION aesthetic non-text images
Benchmarks: MARIO-7M-Eval MARIO-TMDB-Eval MARIO-OpenLibrary-Eval
Metrics: Average Precision Average Recall F1 Accuracy CLIPScore t-SNE font diversity
Baselines: LDM ControlNet TextDiffuser GlyphControl DeepFloyd
Builds on: latent diffusion model TextDiffuser GLASS ControlNet
Closest papers by meaning
- Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model (AAAI 2024 · similarity 0.91)
- DreamText: High Fidelity Scene Text Synthesis (CVPR 2025 · similarity 0.91)
- AnyText: Multilingual Visual Text Generation and Editing (ICLR 2024 · similarity 0.89)
- DesignDiffusion: High-Quality Text-to-Design Image Generation with Diffusion Models (CVPR 2025 · similarity 0.89)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.89)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.