Research map / Image generation / Text-to-image diffusion
DreamText: High Fidelity Scene Text Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamText, a diffusion-based scene text synthesis method that uses latent character masks from cross-attention maps and alternates between optimizing a joint text encoder/U-Net and re-estimating character attention, with warm-up supervision from segmentation masks.
Problem
Scene text synthesis with diffusion models suffers from character repetition, absence, and distortion, especially in polystylistic scenes, due to a single-font character encoder and rigid, noisy character segmentation supervision.
Approach
Reconstruct diffusion training with latent character masks derived from cross-attention maps, used in a heuristic alternate optimization that jointly trains the character text encoder (with masked diffusion, cross-attention, cross-modal alignment and character ID losses) and the U-Net, plus a balanced supervision that drops segmentation-mask guidance after 25k warm-up steps.
Key results
- DreamText reaches average SeqAcc 0.940 (recon 0.95 / editing 0.94 on ICDAR13-level tables, 0.93 overall recon per Table 1 reports) versus UDiffText's 0.94 recon / 0.91 editing on ICDAR13 averages.
- FID improves from 15.79 (UDiffText) to 12.13 for DreamText, and LPIPS from 0.0564 to 0.0328.
- Ablation shows adding all losses raises average SeqAcc from 0.218 (base) to 0.940; 25k warm-up steps gives best trade-off.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DreamText instead of UDiffText | Scene text synthesis · ICDAR13(8ch) reconstruction | SeqAcc: 0.94 -> 0.95 | Helped |
| Use DreamText instead of UDiffText | Scene text synthesis · ICDAR13 reconstruction | SeqAcc: 0.91 -> 0.94 | Helped |
| Use DreamText instead of UDiffText | Scene text synthesis · TextSeg reconstruction | SeqAcc: 0.93 -> 0.96 | Helped |
| Use DreamText instead of UDiffText | Scene text synthesis · LAION-OCR reconstruction | SeqAcc: 0.90 -> 0.93 | Helped |
| Use DreamText instead of UDiffText | Scene text synthesis · ICDAR13(8ch) editing | SeqAcc: 0.84 -> 0.87 | Helped |
| Use DreamText instead of UDiffText | Scene text synthesis · ICDAR13 editing | SeqAcc: 0.83 -> 0.89 | Helped |
| Use DreamText instead of UDiffText | Scene text synthesis · TextSeg editing | SeqAcc: 0.84 -> 0.91 | Helped |
| Use DreamText instead of UDiffText | Scene text synthesis · LAION-OCR editing | SeqAcc: 0.78 -> 0.88 | Helped |
| Use DreamText instead of UDiffText | Scene text synthesis · Text images | FID: 15.79 -> 12.13 | Helped |
| Use DreamText instead of UDiffText | Scene text synthesis · Text images | LPIPS: 0.0564 -> 0.0328 | Helped |
| Add masked diffusion loss (ablation) | DreamText training objective · Average across evaluation datasets | SeqAcc reconstruction: 0.218 -> 0.425 | Helped |
| Add cross-attention loss (ablation) | DreamText training objective · Average across evaluation datasets | SeqAcc reconstruction: 0.425 -> 0.698 | Helped |
Surprising findings
- Masked diffusion and cross-attention losses alone are insufficient; the cross-modal alignment and character ID losses contribute large gains (average SeqAcc 0.698 to 0.940).
Limitations
- Relies on segmentation masks from datasets for warm-up supervision, which are only available for SynthText and LAION-OCR.
- Latent character masks are noisy, causing under- or over-segmentation of character regions.
Experimental setup
Datasets: SynthText LAION-OCR ICDAR13 TextSeg
Benchmarks: ICDAR13 TextSeg LAION-OCR SynthText
Metrics: SeqAcc (reconstruction) SeqAcc (editing) FID LPIPS mIoU
Baselines: SD-Inpainting DiffSTE MOSTEL TextDiffuser AnyText UDiffText
Builds on: Stable Diffusion UDiffText TextDiffuser cross-attention loss (Attend-and-Excite)
Closest papers by meaning
- Layout-Agnostic Scene Text Image Synthesis with Diffusion Models (CVPR 2024 · similarity 0.91)
- Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model (AAAI 2024 · similarity 0.89)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.89)
- TextNeRF: A Novel Scene-Text Image Synthesis Method based on Neural Radiance Fields (CVPR 2024 · similarity 0.89)
- TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control (NeurIPS 2024 · similarity 0.88)
- LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.