Research map / Architectures and efficiency / Attention and transformers
ViTEraser: Harnessing the Power of Vision Transformers for Scene Text Removal with SegMIM Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViTEraser, a single-stage ViT encoder-decoder for scene text removal, pretrained with SegMIM (text box segmentation on the encoder, masked image modeling on the decoder) using scene text detection data.
Problem
Scene text removal relies on complex two-stage or multi-step pipelines sensitive to text localization accuracy, and CNN-based one-stage methods suffer from artifacts and incomplete erasure; ViTs are underexplored for STR and large STR models overfit given scarce training data.
Approach
A pure ViT (Swinv2-based) encoder-decoder with patch embedding and patch splitting layers performs one-stage text erasing without masks, and SegMIM pretrains the encoder on text box segmentation and the decoder on masked image modeling using large scene text detection datasets.
Key results
- ViTEraser-Base with SegMIM reaches 37.11 dB PSNR on SCUT-EnsText and 0.389% recall, 0.768% f-measure (detection-eval), the best among compared methods.
- SegMIM improves PSNR of ViTEraser-Tiny, Small, and Base by 0.48, 0.53, and 0.79 dB on SCUT-EnsText.
- On Tampered-IC13, ViTEraser-Tiny with ContourNet achieves mF 77.02 versus 76.66 for S3R + ContourNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain ViTEraser with SegMIM (ablation) | ViTEraser-Tiny · SCUT-EnsText | PSNR: 36.32 -> 36.80 | Helped |
| Pretrain ViTEraser with SegMIM (ablation) | ViTEraser-Small · SCUT-EnsText | PSNR: 36.55 -> 37.08 | Helped |
| Pretrain ViTEraser with SegMIM (ablation) | ViTEraser-Base · SCUT-EnsText | PSNR: 36.32 -> 37.11 | Helped |
| Pretrain ViTEraser with SegMIM (ablation) | ViTEraser-Tiny · SCUT-Syn | PSNR: 42.24 -> 42.40 | Helped |
| Pretrain ViTEraser with SegMIM (ablation) | ViTEraser-Small · SCUT-Syn | PSNR: 42.45 -> 42.66 | Helped |
| Pretrain ViTEraser with SegMIM (ablation) | ViTEraser-Base · SCUT-Syn | PSNR: 42.53 -> 42.97 | Helped |
| Use a CNN encoder with an added Transformer encoder (ablation) | CNN-based STR architecture · SCUT-EnsText | PSNR: 35.05 -> 34.85 | Hurt |
| Add a Transformer decoder before the deconvolution decoder (ablation) | Swinv2-Tiny encoder + Deconv · SCUT-EnsText | PSNR: 36.06 -> 35.92 | Hurt |
| Replace the decoder with an MLP decoder (ablation) | Swinv2-Tiny encoder · SCUT-EnsText | PSNR: 36.06 -> 26.18 | Hurt |
| Use ViT blocks in both the encoder and decoder instead of a CNN encoder and deconvolution decoder (ablation) | STR architecture · SCUT-EnsText | PSNR: 35.05 -> 36.32 | Helped |
| Increase ViTEraser scale from Swinv2-Tiny to Swinv2-Small (ablation) | ViTEraser-Swinv2 · SCUT-EnsText | PSNR: 36.32 -> 36.55 | Helped |
| Increase ViTEraser scale from Swinv2-Small to Swinv2-Base (ablation) | ViTEraser-Swinv2 · SCUT-EnsText | PSNR: 36.55 -> 36.32 | Hurt |
Surprising findings
- Inserting Transformer modules into CNN encoders/decoders did not improve STR results over the pure CNN baseline.
- Largest PVT and Swinv2 scales underperform smaller ones without pretraining, attributed to overfitting.
Limitations
- Detection-eval and speed comparisons depend on external CRAFT detector and post-processing choices
- Large-scale models overfit on limited STR data without pretraining
- Pretraining on detection data requires box annotations and the mask token must be removed via finetuning
Experimental setup
Datasets: SCUT-EnsText SCUT-Syn ICDAR2013 ICDAR2015 MLT2017 ArT LSVT ReCTS TextOCR ImageNet-1k Tampered-IC13
Benchmarks: SCUT-EnsText SCUT-Syn Tampered-IC13
Metrics: PSNR MSSIM MSE AGE pEPs pCEPs FID recall precision f-measure
Baselines: Pix2pix STE EnsNet MTRNet++ EraseNet SSTE PSSTRNet CTRNet GaRNet MBE PEN PERT
Builds on: Swin Transformer V2 SimMIM EraseNet PVT
Closest papers by meaning
- OTE: Exploring Accurate Scene Text Recognition Using One Token (CVPR 2024 · similarity 0.87)
- CLIP is Almost All You Need: Towards Parameter-Efficient Scene Text Retrieval without OCR (CVPR 2025 · similarity 0.87)
- DreamText: High Fidelity Scene Text Synthesis (CVPR 2025 · similarity 0.86)
- Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP (NeurIPS 2024 · similarity 0.85)
- ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing (NeurIPS 2026 · similarity 0.85)
- ConText: Driving In-context Learning for Text Removal and Segmentation (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.