Research map / Image generation / Text-to-image diffusion
How Control Information Influences Multilingual Text Image Generation and Editing?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes how ControlNet control information (input encoding, denoising-stage role, output frequency features) affects multilingual visual text generation and editing, and proposes TextGen with Fourier-enhanced control blocks, a two-stage global/detail pipeline, and a lightweight 2.5M-image dataset TG-2M.
Problem
Visual text generation with ControlNet-based diffusion models struggles with fine-grained, sparse glyph control conditions; the role of control information across input, denoising stages, and output features is not well understood.
Approach
Treats glyph control as sparse, high-frequency information: uses Fourier Enhancement Convolution to process glyph inputs in spatial and frequency domains, splits control into a global stage (large timesteps) and a detail stage (small timesteps, masked background), and rescales control features in frequency space at inference (FreeU-style balancing), enabling a unified generation/editing paradigm.
Key results
- On AnyWord, TextGen reaches 73.36% English ACC and 67.92% Chinese ACC, versus 65.88% and 66.34% for AnyText-v1.0 (trained on 3.5M).
- Compared with its own baseline trained on TG-2M (2.5M images), TextGen gains about 9.1 points English ACC and 2.9 points Chinese ACC.
- Ablation on 200k images: adding FEC, global perception, two-stage, and inference Fourier enhancement raises English ACC from 49.51 to 60.18 and Chinese ACC from 31.50 to 61.42.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train TextGen with FEC instead of the configuration without components (ablation) | TextGen · Ablation English | ACC: 49.51 -> 50.90 | Helped |
| Train TextGen with FEC instead of the configuration without components (ablation) | TextGen · Ablation Chinese | ACC: 31.50 -> 56.98 | Helped |
| Add global perception to FEC (ablation) | TextGen · Ablation English | ACC: 50.90 -> 52.24 | Helped |
| Add global perception to FEC (ablation) | TextGen · Ablation Chinese | ACC: 56.98 -> 58.60 | Helped |
| Add two-stage generation to FEC and global perception (ablation) | TextGen · Ablation English | ACC: 52.24 -> 53.03 | Helped |
| Add two-stage generation to FEC and global perception (ablation) | TextGen · Ablation Chinese | ACC: 58.60 -> 60.47 | Helped |
| Add inference Fourier enhancement (ablation) | TextGen · Ablation English | ACC: 53.03 -> 60.18 | Helped |
| Add inference Fourier enhancement (ablation) | TextGen · Ablation Chinese | ACC: 60.47 -> 61.42 | Helped |
| Use TextGen rather than the TG-2M-trained baseline | AnyWord English · AnyWord English | ACC: 64.26±0.51 -> 73.36±0.15 | Helped |
| Use TextGen rather than the TG-2M-trained baseline | AnyWord Chinese · AnyWord Chinese | ACC: 65.02±0.11 -> 67.92±0.28 | Helped |
| Use TextGen rather than the TG-2M-trained baseline | AnyWord English · AnyWord English | FID: 41.65±2.84 -> 40.37±1.71 | Hurt |
Surprising findings
- Control in late denoising steps remains important for text, unlike general diffusion where late control contributes little.
- FID is worse than AnyText-v1.0 on AnyWord, which the authors attribute to a distribution gap between training data and the benchmark rather than lower visual quality.
Limitations
- VAE of latent diffusion limits fine-grained generation of small or complex text.
- CLIP text encoder has limited ability to understand text and long captions.
- Higher FID than AnyText-v1.0 on AnyWord, with an acknowledged train/benchmark distribution gap.
Experimental setup
Datasets: TG-2M MARIO-10M Wukong TextSeg ArT LSVT MLT ReCTS AnyWord
Benchmarks: AnyWord
Metrics: ACC NED FID
Baselines: ControlNet GlyphControl TextDiffuser AnyText-v1.0 AnyText baseline trained on TG-2M
Builds on: ControlNet FreeU AnyText Stable Diffusion 1.5
Closest papers by meaning
- AnyText: Multilingual Visual Text Generation and Editing (ICLR 2024 · similarity 0.86)
- Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance (NeurIPS 2024 · similarity 0.85)
- FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation (ICML 2025 · similarity 0.85)
- Rich Human Feedback for Text-to-Image Generation (CVPR 2024 · similarity 0.85)
- Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following (CVPR 2024 · similarity 0.85)
- Layout-Agnostic Scene Text Image Synthesis with Diffusion Models (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.