Research map / Image generation / Text-to-image diffusion
AnyText: Multilingual Visual Text Generation and Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AnyText, a ControlNet-based diffusion model for multilingual visual text generation and editing that fuses OCR stroke embeddings with caption embeddings and uses a text perceptual loss, plus the AnyWord-3M dataset and AnyText-benchmark.
Problem
Diffusion text-to-image models produce blurred, unreadable or incorrect visual text, especially for non-Latin scripts such as Chinese, and lack large annotated multilingual text-image data.
Approach
Combine an auxiliary latent module (glyph, position, masked image) fed to a trainable TextControlNet copy with a text embedding module that replaces placeholder token embeddings with PP-OCRv3 recognition features of rendered glyph lines, plus a text perceptual loss computed by OCR features in image space on text regions only.
Key results
- On AnyText-benchmark Chinese, AnyText-v1.1 reaches Sen. ACC 0.6923 and NED 0.8396 vs ControlNet 0.3620 and TextDiffuser 0.0605.
- On English, AnyText-v1.1 achieves Sen. ACC 0.7239 and FID 33.54, beating TextDiffuser (0.5921, 41.31).
- In ablation, OCR-based text embedding raises Chinese Sen. ACC from 0.2024 to 0.4595, and perceptual loss with lambda=0.01 adds about 4 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| enable the editing branch (ablation) | AnyText · Chinese ablation benchmark | Sen. ACC: 0.2024 -> 0.1552 | Hurt |
| replace direct CLIP caption encoding with a CLIP vision encoder (vit) for text embeddings (ablation) | AnyText · Chinese ablation benchmark | Sen. ACC: 0.2024 -> 0.1416 | Hurt |
| replace direct CLIP caption encoding with a trainable convolutional module for text embeddings (ablation) | AnyText · Chinese ablation benchmark | Sen. ACC: 0.2024 -> 0.1864 | Hurt |
| use PP-OCRv3 for text embeddings instead of direct CLIP caption encoding (ablation) | AnyText · Chinese ablation benchmark | Sen. ACC: 0.2024 -> 0.4595 | Helped |
| add the explicit position condition (ablation) | AnyText with PP-OCRv3 text embeddings · Chinese ablation benchmark | Sen. ACC: 0.4472 -> 0.4595 | Helped |
| add text perceptual loss with λ=0.003 (ablation) | AnyText with PP-OCRv3 text embeddings · Chinese ablation benchmark | Sen. ACC: 0.4595 -> 0.4848 | Helped |
| add text perceptual loss with λ=0.01 (ablation) | AnyText with PP-OCRv3 text embeddings · Chinese ablation benchmark | Sen. ACC: 0.4595 -> 0.4996 | Helped |
| add text perceptual loss with λ=0.03 (ablation) | AnyText with PP-OCRv3 text embeddings · Chinese ablation benchmark | Sen. ACC: 0.4595 -> 0.4659 | Helped |
| use AnyText instead of the strongest baseline for English sentence accuracy | AnyText-v1.1 · AnyText-benchmark English | Sen. ACC: 0.5921 -> 0.7239 | Helped |
| use AnyText instead of the strongest baseline for English normalized edit distance | AnyText-v1.1 · AnyText-benchmark English | NED: 0.8015 -> 0.8760 | Helped |
| use AnyText instead of the strongest baseline for English FID | AnyText-v1.1 · AnyText-benchmark English | FID: 37.84 -> 33.54 | Helped |
| use AnyText instead of the strongest baseline for Chinese sentence accuracy | AnyText-v1.1 · AnyText-benchmark Chinese | Sen. ACC: 0.3620 -> 0.6923 | Helped |
Surprising findings
- ControlNet with canny control scores poorly on whole-image OCR metrics because it generates pseudo-text in backgrounds, though it performs well inside the specified text regions.
- CLIP vision encoder features used as text embeddings performed worse than a simple caption-only CLIP text encoder baseline in the ablation.
Limitations
- Generating extremely small fonts is not addressed (stated as future work).
- Text generation with controllable attributes is not explored.
- Pseudo-text in backgrounds and watermarks remain issues, with OCR annotation noise affecting metrics.
Experimental setup
Datasets: AnyWord-3M Noah-Wukong LAION-400M OCR recognition datasets
Benchmarks: AnyText-benchmark
Metrics: Sentence Accuracy Normalized Edit Distance FID
Baselines: ControlNet TextDiffuser GlyphControl GlyphDraw SD-XL 1.0 DALL-E 2 DeepFloyd IF
Builds on: ControlNet Stable Diffusion PP-OCRv3 BLIP-2
Closest papers by meaning
- Layout-Agnostic Scene Text Image Synthesis with Diffusion Models (CVPR 2024 · similarity 0.89)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.88)
- Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model (AAAI 2024 · similarity 0.88)
- DreamText: High Fidelity Scene Text Synthesis (CVPR 2025 · similarity 0.88)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.87)
- Harmonizing Visual Text Comprehension and Generation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.