Research map / Image generation / Editing and style transfer
GlyphMastero: A Glyph Encoder for High-Fidelity Scene Text Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GlyphMastero, a trainable hierarchical glyph encoder built on PaddleOCR-v4 features (glyph attention plus FPN) that conditions a latent inpainting diffusion model for scene text editing with stroke-level accuracy.
Problem
Diffusion-based scene text editing often produces distorted or unrecognizable characters, especially complex Chinese characters, because existing OCR-feature guidance fails to capture hierarchical stroke-to-character-to-line structure.
Approach
A trainable glyph encoder uses glyph attention modules to model cross-level interactions between per-character local OCR features and a global text-line feature, and an FPN fuses multi-scale OCR backbone features; the resulting embedding conditions the inpainting UNet via cross-attention and is trained jointly with it.
Key results
- Overall sentence accuracy is 18.02% higher than AnyText and 48.14% higher than DiffUTE (English 0.8170, Chinese 0.7301 for the method).
- Text-region FID is 53.28% lower than AnyText on average (Chinese FID 11.89 vs 24.90; English 4.61 vs 10.43).
- English-only comparison: best sentence accuracy 0.8170 and CER 0.0741, though TextCtrl has marginally better FID and LPIPS (4.15 and 0.0425 vs 4.61 and 0.0545).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GlyphMastero instead of AnyText | Scene text editing · AnyText-Eval English and Chinese | Sen.Acc: 18.02% higher than AnyText | Helped |
| Use GlyphMastero instead of AnyText | Scene text editing · AnyText-Eval English and Chinese | FID: 53.28% lower than AnyText | Helped |
| Use GlyphMastero instead of TextCtrl | English scene text editing · English-only comparison | Sen.Acc: 0.7654 -> 0.8170 | Helped |
| Use GlyphMastero instead of TextCtrl | English scene text editing · English-only comparison | FID: 4.15 -> 4.61 | Hurt |
| Add multi-scale FPN fusion of backbone features (ablation) | GlyphMastero · Ablation subset English | Sen.Acc: 0.4536 -> 0.5494 | Helped |
| Add multi-scale FPN fusion of backbone features (ablation) | GlyphMastero · Ablation subset Chinese | Sen.Acc: 0.3698 -> 0.5120 | Helped |
| Add backbone glyph attention module Tb (ablation) | GlyphMastero · Ablation subset English | Sen.Acc: 0.5065 -> 0.5494 | Helped |
| Add backbone glyph attention module Tb (ablation) | GlyphMastero · Ablation subset Chinese | Sen.Acc: 0.4271 -> 0.5120 | Helped |
| Use glyph attention Tn instead of local neck features ln alone (ablation) | GlyphMastero · Ablation subset English | Sen.Acc: 0.3263 -> 0.5494 | Helped |
| Use glyph attention Tn instead of local neck features ln alone (ablation) | GlyphMastero · Ablation subset Chinese | Sen.Acc: 0.2735 -> 0.5120 | Helped |
| Use glyph attention Tn instead of global neck features gn (ablation) | GlyphMastero · Ablation subset English | Sen.Acc: 0.1003 -> 0.5494 | Helped |
| Use glyph attention Tn instead of global neck features gn (ablation) | GlyphMastero · Ablation subset Chinese | Sen.Acc: 0.0719 -> 0.5120 | Helped |
Surprising findings
- Removing both the FPN and the backbone glyph attention module yielded better performance than removing the FPN alone, which the authors attribute to the FPN refining features for the backbone attention module.
- Using only the global neck feature as guidance, the approach used by AnyText, yielded the poorest accuracy in ablation.
Limitations
- Accuracy on long text still lags behind short text.
- Training data limited to 512x512 resolution and constrained by the base latent diffusion model.
Experimental setup
Datasets: AnyWord-3M AnyText-Eval
Benchmarks: AnyText-Eval TextCtrl ScenePair
Metrics: Sentence accuracy CER FID LPIPS
Baselines: DiffUTE AnyText SRNet MOSTEL DiffSTE TextDiffuser TextCtrl
Builds on: latent diffusion models DiffUTE AnyText ControlNet Feature Pyramid Network
Closest papers by meaning
- GlyphSR: A Simple Glyph-Aware Framework for Scene Text Image Super-Resolution (AAAI 2025 · similarity 0.90)
- GlyphAnchor: Enhancing Visual Text Rendering via Position-Anchored Glyph Priors (NeurIPS 2026 · similarity 0.89)
- TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control (NeurIPS 2024 · similarity 0.89)
- GlyphDraw2: Automatic Generation of Complex Glyph Posters with Diffusion Models and Large Language Models (AAAI 2025 · similarity 0.88)
- SteerVTE: Seamless Video Text Editing with Style and Glyph Control (NeurIPS 2026 · similarity 0.87)
- DreamText: High Fidelity Scene Text Synthesis (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.