Research map / Image generation / Editing and style transfer
Choose What You Need: Disentangled Representation Learning for Scene Text Recognition Removal and Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DARLING, a pre-training framework that disentangles scene text images into style and content features using synthetic image pairs, and applies the pre-trained model to scene text recognition, editing, and removal.
Problem
Existing representation learning for scene text uses a single tightly coupled feature for both discriminative (recognition) and generative (editing, removal) tasks, which is sub-optimal because the tasks need different information.
Approach
Split ViT tokens into style and content halves; supervise content with a recognition loss, align style features across image pairs sharing style, and reconstruct the counterpart image from style features plus a text prompt, so the features disentangle and a single multi-task decoder serves all three tasks.
Key results
- Scene text recognition averages 80.2% on the 6 common benchmarks plus Union14M-Benchmark-style evaluation, with DARLING pre-trained on 4M synthetic pairs outperforming pre-training methods that use ~10M real images.
- On Tamper-Scene real editing, SeqAcc improves by 10.94% over prior state of the art, and ClassAcc improves by 3%.
- On SCUT-EnsText removal, DARLING reaches MSE 0.0005, PSNR 38.85, and SSIM 98.25, the best across all metrics reported.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use disentangled pre-training | DARLING STR model · Common benchmarks | accuracy: 94.8 -> 95.6 | Helped |
| use disentangled pre-training | DARLING STR model · Union14M-Benchmark | accuracy: 75.8 -> 80.2 | Helped |
| use DARLING instead of MOSTEL | scene text editing · Tamper-Scene | SeqAcc: 66.54 -> 70.85 | Helped |
| use DARLING instead of MOSTEL | scene text editing · STE-10k | SeqAcc: 50.71 -> 61.00 | Helped |
| use DARLING instead of the baseline without disentangled pre-training | scene text removal · SCUT-EnsText | PSNR: 33.11 -> 38.85 | Helped |
| add gated injection (ablation) | DARLING scene text editing · Tamper-Syn2k | MSE: 0.0136 -> 0.0120 | Helped |
| add gated injection (ablation) | DARLING scene text editing · Tamper-Syn2k | PSNR: 19.56 -> 20.80 | Helped |
| add gated injection (ablation) | DARLING scene text editing · Tamper-Syn2k | SSIM: 54.02 -> 60.07 | Helped |
| add gated injection (ablation) | DARLING scene text editing · STE-10k | MSE: 0.0148 -> 0.0100 | Helped |
| add gated injection (ablation) | DARLING scene text editing · STE-10k | PSNR: 20.46 -> 21.77 | Helped |
| add gated injection (ablation) | DARLING scene text editing · STE-10k | SSIM: 57.69 -> 59.95 | Helped |
Surprising findings
- A single pre-trained model with no discriminator or extra pre-training performs editing and removal competitively, matching or beating methods that require more supervision.
Limitations
- Synthetic pre-training data has a domain gap with real scene text.
- Performance is slightly weaker on multi-word images.
- Unlabeled real data cannot be used to pre-train the decoder in the current framework.
Experimental setup
Datasets: TSE-4M Union-L SCUT-EnsText SynthText MJSynth Tamper-Syn2k Tamper-Scene
Benchmarks: IIIT5K IC13 SVT IC15 SVTP CUTE Union14M-Benchmark WOST HOST WordArt Tamper-Syn2k Tamper-Scene
Metrics: accuracy MSE PSNR SSIM FID SeqAcc ClassAcc
Baselines: MAERec-S CCD PARSeq ABINet MOSTEL SRNet SwapText EraseNet CTRNet ViTEraser-Tiny
Builds on: MAERec SimAN SynthText SRNet
Closest papers by meaning
- Recognition-Synergistic Scene Text Editing (CVPR 2025 · similarity 0.85)
- Disentangled 3D Scene Generation with Layout Learning (ICML 2024 · similarity 0.81)
- CLIP in Mirror: Disentangling text from visual images through reflection (NeurIPS 2024 · similarity 0.81)
- Boosting Semi-Supervised Scene Text Recognition via Viewing and Summarizing (NeurIPS 2024 · similarity 0.81)
- DisenBooth: Identity-Preserving Disentangled Tuning for Subject-Driven Text-to-Image Generation (ICLR 2024 · similarity 0.80)
- An Item Is Worth a Prompt: Versatile Image Editing with Disentangled Control (AAAI 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.