Research map / Image generation / Editing and style transfer
TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TextCtrl, a diffusion-based scene text editing method that uses disentangled style and glyph-structure priors plus a glyph-adaptive mutual self-attention at inference, and releases the ScenePair real-world image-pair benchmark.
Problem
Scene text editing must change text content while preserving text style and background; GAN methods generalize poorly and diffusion methods show style deviations and spelling errors.
Approach
Explicit style features (from a multi-task pre-trained ViT encoder for color, font, removal, segmentation) and glyph-structure features (character-level transformer aligned to a frozen recognizer with font-variance augmentation) guide a Stable Diffusion generator, and a glyph-adaptive mutual self-attention blends source-branch and edit-branch keys/values during DDIM sampling, weighted by recognize
Key results
- On ScenePair cropped text images, TextCtrl reaches SSIM 37.56 (x10^-2), PSNR 14.99, MSE 4.47 and FID 43.78, the best on all four metrics.
- On ScenePair text rendering accuracy, TextCtrl achieves 84.67% ACC and 0.936 NED, versus 51.48% ACC for the next best (TextDiffuser).
- Replacing ControlNet with the pre-trained style encoder S improves FID from 58.30 to 43.78 in the ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TextCtrl instead of the strongest reported baseline | Scene text editing · ScenePair cropped text image | SSIM: 30.73 -> 37.56 | Helped |
| use TextCtrl instead of the strongest reported baseline | Scene text editing · ScenePair cropped text image | PSNR: 14.46 -> 14.99 | Helped |
| use TextCtrl instead of the strongest reported baseline | Scene text editing · ScenePair cropped text image | FID: 49.19 -> 43.78 | Helped |
| use TextCtrl instead of the strongest reported baseline | Scene text editing · ScenePair | ACC (%): 51.48 -> 84.67 | Helped |
| use TextCtrl instead of the strongest reported baseline | Scene text editing · TamperScene | ACC (%): 76.79 -> 74.17 | Hurt |
| add font-variance augmentation to glyph structure pre-training (ablation) | TextCtrl · ScenePair | ACC (%): 76.08 -> 84.67 | Helped |
| replace CLIP text encoder with the glyph structure encoder (ablation) | TextCtrl · ScenePair | ACC (%): 13.98 -> 84.67 | Helped |
| add style disentanglement pre-training (ablation) | TextCtrl · ScenePair cropped text image | SSIM: 0.3130 -> 0.3756 | Helped |
| replace ControlNet style injection with the pre-trained style encoder (ablation) | TextCtrl · ScenePair cropped text image | FID: 58.30 -> 43.78 | Helped |
| add GaMuSa inference control (ablation) | TextCtrl · ScenePair cropped text image | SSIM: 0.3126 -> 0.3756 | Helped |
| replace direct sampling with GaMuSa (ablation) | TextCtrl · ScenePair cropped text image | FID: 51.35 -> 43.78 | Helped |
| replace MasaCtrl with GaMuSa (ablation) | TextCtrl · ScenePair cropped text image | FID: 49.53 -> 43.78 | Helped |
Surprising findings
- On TamperScene, MOSTEL achieves higher ACC (76.79%) than TextCtrl (74.17%), where TextCtrl is second best.
- The CLIP text encoder yields only 13.98% ACC, showing that generic prompt embeddings struggle with glyph-accurate rendering.
Limitations
- Arbitrary-shape (curved or circular) text editing is unreliable because geometric priors are hard to inject into diffusion models.
- Existing visual quality metrics are pixel- or feature-based and sub-optimal for text style coherency; ScenePair needs paired ground truth.
- Inference relies on a detected or user-provided text region and perspective cropping before editing.
Experimental setup
Datasets: synthetic paired text images (200k, following AnyText data pipeline) ICDAR 2013 HierText MLT 2017 TamperScene
Benchmarks: ScenePair ScenePair (Random) TamperScene
Metrics: SSIM PSNR MSE FID ACC NED
Baselines: SRNet MOSTEL DiffSTE TextDiffuser AnyText MasaCtrl
Builds on: Stable Diffusion MasaCtrl ControlNet DDIM inversion AnyText
Closest papers by meaning
- GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editing (CVPR 2025 · similarity 0.89)
- SteerVTE: Seamless Video Text Editing with Style and Glyph Control (NeurIPS 2026 · similarity 0.88)
- DreamText: High Fidelity Scene Text Synthesis (CVPR 2025 · similarity 0.88)
- Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance (NeurIPS 2024 · similarity 0.88)
- Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Model (AAAI 2024 · similarity 0.88)
- AdapEdit: Spatio-Temporal Guided Adaptive Editing Algorithm for Text-Based Continuity-Sensitive Image Editing (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.