Research map / Image generation / Text-to-image diffusion
Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Training-free multilingual scene text image generation by steering Stable Diffusion + ControlNet (Canny) with a localized cross-attention constraint and contrastive positive/negative image-level prompts.
Problem
Diffusion models struggle to generate accurate scene text, especially for low-resource languages (e.g., Arabic, Thai, Kazakh) that are underrepresented in training data, and lack control over text placement.
Approach
Render the target text as a sketch image, extract Canny edges as a ControlNet condition, constrain cross-attention maps of text-region keywords (e.g., "sign", "billboard") to the text bounding box, and apply contrastive image-level prompts (a 1-pixel box as positive prompt, a blank white image as negative prompt) in classifier-free-guidance-style combination, all without training.
Key results
- Full model achieves CLIPScore 0.8108 and accuracy 35.48 on the ablation set, versus 31.42 without the localized attention constraint and 10.39 without the negative image-level prompt.
- On Arabic, Thai, Russian and Chinese, Diff-Text reaches OCR accuracy of 33.13, 38.41, 39.29 and 32.40 versus at most 9.79 for ControlNet and 0 for Stable Diffusion, DeepFloyd and TextDiffuser.
- On English, Diff-Text reaches accuracy 61.03 versus 43.91 for TextDiffuser, with edit accuracy 89.52 versus 84.84.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add localized attention constraint (ablation) | Diff-Text · Ablation evaluation | CLIP: 0.8065 -> 0.8108 | Helped |
| add localized attention constraint (ablation) | Diff-Text · Ablation evaluation | Accuracy: 31.42 -> 35.48 | Helped |
| add localized attention constraint (ablation) | Diff-Text · Ablation evaluation | Edit accuracy: 74.30 -> 77.22 | Helped |
| add positive image-level prompt (ablation) | Diff-Text · Ablation evaluation | CLIP: 0.7935 -> 0.8108 | Helped |
| add positive image-level prompt (ablation) | Diff-Text · Ablation evaluation | Accuracy: 27.68 -> 35.48 | Helped |
| add positive image-level prompt (ablation) | Diff-Text · Ablation evaluation | Edit accuracy: 70.92 -> 77.22 | Helped |
| add negative image-level prompt (ablation) | Diff-Text · Ablation evaluation | CLIP: 0.7718 -> 0.8108 | Helped |
| add negative image-level prompt (ablation) | Diff-Text · Ablation evaluation | Accuracy: 10.39 -> 35.48 | Helped |
| add negative image-level prompt (ablation) | Diff-Text · Ablation evaluation | Edit accuracy: 50.95 -> 77.22 | Helped |
| use Diff-Text instead of the strongest baseline | Scene text generation · English | CLIPScore: 0.8666 -> 0.8649 | Hurt |
| use Diff-Text instead of the strongest baseline | Scene text generation · Arabic | Accuracy: 3.291 -> 33.13 | Helped |
| use Diff-Text instead of the strongest baseline | Scene text generation · Chinese | Edit_accuracy: 26.81 -> 68.75 | Helped |
Surprising findings
- Negative image-level prompt removal causes the largest drop (accuracy 35.48 to 10.39), which the authors attribute to edge images being misread as object outlines.
- Authors note the OCR tool underestimates accuracy: on sketch images, English accuracy is 92.80 and Chinese only 60.85.
Limitations
- Struggles to generate small-scale scene text and precise text color control.
- Generated images occasionally include unintended textual elements, such as unexpected English text.
- Text region bounding boxes and attention wordlists are manually specified or randomly generated rather than chosen automatically.
Experimental setup
Datasets: multilingual vocabularies from Zhang et al. and Xie et al.
Benchmarks: custom multilingual scene text generation set (English, Chinese, Arabic, Thai, Russian; 3000 words per language)
Metrics: CLIPScore OCR accuracy normalized edit distance
Baselines: Stable Diffusion DeepFloyd TextDiffuser ControlNet
Builds on: Stable Diffusion ControlNet classifier-free guidance
Closest papers by meaning
- Layout-Agnostic Scene Text Image Synthesis with Diffusion Models (CVPR 2024 · similarity 0.91)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.90)
- DreamText: High Fidelity Scene Text Synthesis (CVPR 2025 · similarity 0.89)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.89)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.89)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.