Research map / Image generation / Editing and style transfer
Recognition-Synergistic Scene Text Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RS-STE, a unified transformer model that jointly performs scene text recognition and editing via a multi-modal parallel decoder, trained with paired synthetic data then cyclic self-supervised fine-tuning on unpaired real images.
Problem
Scene text editing must replace text content in scene images while preserving style, but existing pipelines explicitly disentangle style and content with complex multi-module designs that generalize poorly to real images lacking paired training data.
Approach
Integrate text recognition and editing in one transformer multi-modal parallel decoder so recognition implicitly separates style from content and enforces content consistency; a twice-cyclic editing procedure (edit to new text, then edit back) gives self-supervision on unpaired real data.
Key results
- On Tamper-Scene, RS-STE reaches 86.12% RecAcc, a 7.32% gain over STEEM.
- Across six STR benchmarks, RS-STE trained on Union14M-L averages 82.9% recognition accuracy on edited images versus 36.8% for MOSTEL on MLT2017 data (about 45% relative improvement cited).
- Using RS-STE edited hard cases as augmentation raises ABINet average accuracy by 2.2% and MAERec-S by 2.5% on Union14M-Benchmark.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RS-STE instead of STEEM | scene text editing · Tamper-Syn2k | RecAcc: 78.80 -> 86.12 | Helped |
| Use RS-STE instead of TextCtrl | scene text editing · Tamper-Scene | RecAcc: 84.67 -> 91.80 | Helped |
| Use RS-STE instead of MOSTEL with MLT2017 training | text editing on STR recognition benchmarks · STR recognition benchmarks average | accuracy: 36.8 -> 81.8 | Helped |
| Use Union14M-L instead of MLT2017 for fine-tuning | RS-STE · STR recognition benchmarks average | accuracy: 81.8 -> 82.9 | Helped |
| Add recognition loss during pre-training (ablation) | RS-STE · Tamper-Syn2k | SSIM: 69.70 -> 72.90 | Helped |
| Add perceptual loss during pre-training (ablation) | RS-STE · Tamper-Syn2k | FID: 70.06 -> 30.29 | Helped |
| Add MSE loss during pre-training (ablation) | RS-STE · Tamper-Syn2k | SSIM: 78.02 -> 72.90 | Hurt |
| Replace intrinsic recognition supervision with external ABINet supervision (ablation) | RS-STE · Tamper-Syn2k | SSIM: 70.71 -> 72.90 | Hurt |
| Add cyclic perceptual loss (ablation) | RS-STE fine-tuned on unpaired real data · Tamper-Scene | RecAcc: 96.23 -> 86.12 | Hurt |
| Add cyclic MSE loss (ablation) | RS-STE fine-tuned on unpaired real data · Tamper-Scene | RecAcc: 83.79 -> 86.12 | Helped |
| Add first-cycle recognition loss (ablation) | RS-STE fine-tuned on unpaired real data · Tamper-Scene | RecAcc: 0.00 -> 86.12 | Helped |
| Add second-cycle recognition loss (ablation) | RS-STE fine-tuned on unpaired real data · Tamper-Scene | RecAcc: 0.00 -> 86.12 | Helped |
Surprising findings
- Removing the recognition losses on the cyclic stage causes RecAcc to collapse to 0%, indicating the model learns an identity mapping without them.
Limitations
- Synthetic ground-truth images in Tamper-Syn2k are sometimes indiscernible, so FID is worse than some baselines.
- Performance on real data still depends on fine-tuning with real images; without it RecAcc on real benchmarks is much lower.
Experimental setup
Datasets: Tamper-train SRNet-Datagen synthetic 4M pairs MLT-2017 Union14M-L Tamper-Syn2k ScenePair Tamper-Scene IIIT 5K-Words ICDAR2013 SVT ICDAR2015 SVTP
Benchmarks: Tamper-Syn2k ScenePair Tamper-Scene IIIT IC13 SVT IC15 SVTP CUTE Union14M-Benchmark
Metrics: MSE PSNR SSIM FID RecAcc word recognition accuracy
Baselines: pix2pix SRNet SwapText MOSTEL DARLING STEEM TextCtrl
Builds on: LDM MOSTEL minGPT SRNet
Closest papers by meaning
- Choose What You Need: Disentangled Representation Learning for Scene Text Recognition Removal and Editing (CVPR 2024 · similarity 0.85)
- Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer (CVPR 2024 · similarity 0.84)
- Boosting Semi-Supervised Scene Text Recognition via Viewing and Summarizing (NeurIPS 2024 · similarity 0.83)
- Text-Driven Image Editing via Learnable Regions (CVPR 2024 · similarity 0.83)
- ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing (NeurIPS 2026 · similarity 0.82)
- Preserve or Modify? Context-Aware Evaluation for Balancing Preservation and Modification in Text-Guided Image Editing (CVPR 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.