Research map / Detection and segmentation / CNN features
UPOCR: Towards Unified Pixel-Level OCR Interface
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UPOCR, a single ViT-based encoder-decoder with learnable task prompts that unifies text removal, text segmentation, and tampered text detection as RGB image-to-image translation, trained with one pixel/feature loss.
Problem
Pixel-level OCR tasks (text removal, text segmentation, tampered text detection) use divergent paradigms, architectures, and losses, making unified research and deployment hard; existing generalist models either target documents or cannot produce pixels.
Approach
Cast diverse pixel-level OCR tasks as RGB image-to-image translation, use one ViT-based encoder-decoder (initialized from ViTEraser-Small) with learnable task prompts added to the shared encoder-decoder feature, and train all tasks with the same pixel (multi-scale L1/smooth L1) and feature (perceptual + style) losses.
Key results
- On SCUT-EnsText text removal, UPOCR reaches 37.14 PSNR and 10.47 FID, beating ViTEraser-Base (37.11 PSNR, 10.15 FID on FID) on image-eval PSNR/MSE.
- On TextSeg, UPOCR achieves 88.76 fgIoU and 94.04 F, above TFT (87.11 fgIoU, 93.10 F).
- On Tampered-IC13, UPOCR achieves 71.71 mIoU and 83.53 mF, outperforming Swin-Uper (69.55 mIoU, 81.80 mF) and Painter (69.26 mIoU) under the same retraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UPOCR instead of ViTEraser-Base | Text removal · SCUT-EnsText | PSNR: 37.11 -> 37.14 | Helped |
| Use UPOCR instead of TFT | Text segmentation · TextSeg | fgIoU: 87.11 -> 88.76 | Helped |
| Use UPOCR instead of Swin-Uper | Tampered text detection · Tampered-IC13 | mIoU: 69.55 -> 71.71 | Helped |
| Insert prompts in encoder and shared feature rather than shared feature only (ablation) | UPOCR · SCUT-EnsText | PSNR: 37.14 -> 36.93 | Hurt |
| Insert prompts in shared feature and decoder rather than shared feature only (ablation) | UPOCR · Tampered-IC13 | mIoU: 71.71 -> 71.13 | Hurt |
| Insert prompts in encoder, shared feature, and decoder rather than shared feature only (ablation) | UPOCR · SCUT-EnsText | PSNR: 37.14 -> 36.86 | Hurt |
| Add text segmentation to text removal training (ablation) | UPOCR · SCUT-EnsText | PSNR: 36.97 -> 36.91 | Hurt |
| Add tampered text detection to text removal training (ablation) | UPOCR · SCUT-EnsText | PSNR: 36.97 -> 37.08 | Helped |
| Add text removal to text segmentation training (ablation) | UPOCR · TextSeg | fgIoU: 88.83 -> 88.64 | Hurt |
| Add tampered text detection to text segmentation training (ablation) | UPOCR · TextSeg | fgIoU: 88.83 -> 89.07 | Helped |
| Add text removal to tampered text detection training (ablation) | UPOCR · Tampered-IC13 | mIoU: 67.73 -> 69.26 | Helped |
| Add text segmentation to tampered text detection training (ablation) | UPOCR · Tampered-IC13 | mIoU: 67.73 -> 70.78 | Helped |
Surprising findings
- Adding tampered text detection data helps text segmentation, while adding text removal data degrades text segmentation.
- OFA's VQGAN-based reconstruction upper bound is already well below UPOCR on text removal, so sequence-to-sequence generalists are limited by the discrete-token decoder.
Limitations
- Tampered-IC13 has only 229 training samples, so the model relies on joint training to perform well.
- Text removal and text segmentation conflict when trained jointly, so task collaboration is not uniformly positive.
- Evaluated on only three pixel-level OCR tasks with per-task training data drawn from fixed benchmarks.
Experimental setup
Datasets: SCUT-EnsText TextSeg Tampered-IC13
Benchmarks: SCUT-EnsText TextSeg Tampered-IC13
Metrics: PSNR MSSIM MSE AGE pEPs pCEPs FID fgIoU F-measure mIoU
Baselines: Pix2pix STE EnsNet MTRNet++ EraseNet ViTEraser-Base CTRNet Painter OFA DeepLabV3+
Builds on: ViTEraser Painter Swin Transformer V2 EraseNet
Closest papers by meaning
- MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios (NeurIPS 2025 · similarity 0.87)
- OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning (NeurIPS 2025 · similarity 0.86)
- PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Tasks (CVPR 2026 · similarity 0.86)
- ODM: A Text-Image Further Alignment Pre-training Approach for Scene Text Detection and Spotting (CVPR 2024 · similarity 0.86)
- ConText: Driving In-context Learning for Text Removal and Segmentation (ICML 2025 · similarity 0.85)
- InstructOCR: Instruction Boosting Scene Text Spotting (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.