Research map / Image generation / Text-to-image diffusion
Text Image Inpainting via Global Structure-Guided Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formulates text image inpainting for blind corruption, builds TII-ST and TII-HT datasets, and proposes GSDM, which predicts a global text segmentation mask and then runs a conditional diffusion model to restore corrupted text images.
Problem
Restoring corrupted real-world text images (graffiti, incomplete signatures) when the corrupted mask is unknown, where corruption breaks global text structure and styles vary widely; this hurts scene text recognition and signature/handwriting recognition.
Approach
A Structure Prediction Module (dilated U-Net) predicts the complete foreground segmentation mask from the corrupted image without a mask; a diffusion Reconstruction Module conditioned on the corrupted image and predicted mask regresses the clean image (x0-prediction) with one-step non-Markov DDIM-style inference.
Key results
- On TII-ST, GSDM reaches 67.48 CRNN / 74.67 ASTER / 73.04 MORAN word accuracy and 33.28 PSNR, vs 60.41 / 70.61 / 70.55 and 28.36 PSNR for TransCNN-HAE.
- On TII-HT, GSDM reaches 69.43 DAN / 66.81 TrOCR-L word accuracy and 32.13 PSNR, vs 47.25 best baseline for TrOCR-L and 24.52 PSNR for CoPaint.
- Single-step inference takes 0.034 s per image on RM, versus 72.85 s for CoPaint and 17.56 s for 1000-step Markov DDPM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add global structure guidance via SPM to RM, predicting image x (ablation) | GSDM · TII-ST | accuracy: 69.40 -> 71.73 | Helped |
| Predict image x instead of noise ε (ablation) | GSDM · TII-ST | accuracy: 56.10 -> 71.73 | Helped |
| Use Non-Markov sampling instead of Markov sampling (ablation) | GSDM reconstruction module · TII-ST | accuracy: 68.21 -> 71.73 | Helped |
| Use 5 inference steps instead of 1 (ablation) | GSDM · TII-ST | accuracy: 71.73 -> 69.38 | Hurt |
| Use 10 inference steps instead of 1 (ablation) | GSDM · TII-ST | accuracy: 71.73 -> 68.96 | Hurt |
| Add segmentation loss Lseg (ablation) | GSDM training objective · TII-ST | accuracy: 69.80 -> 70.00 | Helped |
| Add character perceptual loss Lcha alongside Lseg (ablation) | GSDM training objective · TII-ST | accuracy: 70.00 -> 70.19 | Helped |
| Add style loss Lsty alongside Lseg (ablation) | GSDM training objective · TII-ST | accuracy: 70.00 -> 70.09 | Helped |
| Add both character perceptual loss Lcha and style loss Lsty (ablation) | GSDM training objective · TII-ST | accuracy: 70.00 -> 71.73 | Helped |
| Add SPM global structure guidance and image-target prediction (ablation) | GSDM · TII-ST | accuracy: 55.35 -> 71.73 | Helped |
| Use GSDM instead of TransCNN-HAE | Text image inpainting · TII-ST | PSNR: 28.36 -> 33.28 | Helped |
| Use GSDM instead of TransCNN-HAE | Text image inpainting · TII-HT | DAN word accuracy: 17.19 -> 69.43 | Helped |
Surprising findings
- Predicting the clean image x0 substantially outperforms predicting noise epsilon in the diffusion reconstruction module (71.73 vs 56.10 accuracy).
- Increasing the number of non-Markov inference steps lowers performance; a single step performs best.
Limitations
- Corrosion forms are synthetic simulations of real damage rather than naturally corrupted images.
- Quick-draw and convex-hull performance trails TransCNN-HAE on some settings (e.g., convex hull on TII-ST).
Experimental setup
Datasets: ICDAR 2013 ICDAR 2015 ICDAR 2017 IAM QD-IMD
Benchmarks: TII-ST TII-HT
Metrics: word accuracy character accuracy PSNR SSIM
Baselines: DDIM CoPaint TransCNN-HAE TSINIT Wang et al. VCNet GDP
Builds on: DDPM DDIM CoPaint TransCNN-HAE TSINIT
Closest papers by meaning
- One Stone with Two Birds: A Null-Text-Null Frequency-Aware Diffusion Models for Text-Guided Image Inpainting (NeurIPS 2025 · similarity 0.89)
- Diffusion-based Blind Text Image Super-Resolution (CVPR 2024 · similarity 0.87)
- Layout-Agnostic Scene Text Image Synthesis with Diffusion Models (CVPR 2024 · similarity 0.86)
- TextCtrl: Diffusion-based Scene Text Editing with Prior Guidance Control (NeurIPS 2024 · similarity 0.86)
- HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models (ICLR 2025 · similarity 0.86)
- MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.