Research map / Vision-language and multimodal / Human and embodied modeling
Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LMIM, a self-supervised scene text recognition method that feeds a differently-styled view of the same text into a linguistic guidance branch whose [cls] feature conditions the MAE-style decoder, and releases a 11M unlabeled Chinese text image dataset.
Problem
Self-supervised scene text recognition learns visual structure well, but masked image modeling and sequence contrastive learning capture limited global linguistic context of characters within a word.
Approach
A dual-branch MIM where a second view with the same text but different visual appearance is encoded with shared weights; an alignment loss on [cls] tokens yields vision-independent linguistic guidance that is fused into the MIM decoder via self- and cross-attention, forcing global context use for reconstruction.
Key results
- Achieves 86.3% average accuracy on Union14M benchmark with 20-epoch pre-training (ViT-S), 2.0% over SSM.
- Reaches 97.0% average accuracy on six common benchmarks (96.7% weighted) with corrected labels.
- Chinese benchmark: 83.6% on Scene and 82.0% on Web, state-of-the-art average 85.5 W-Avg.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the linguistic guidance branch and alignment loss to masked reconstruction (ablation) | LMIM · Union14M ablation | Avg accuracy: 77.7 -> 81.2 | Helped |
| Add the alignment loss (ablation) | LMIM with linguistic guidance · Union14M ablation | Avg accuracy: 80.6 -> 81.2 | Helped |
| Use medium rather than strong or weak augmentation for the guidance view (ablation) | LMIM · Union14M ablation | Avg accuracy: 79.3 / 79.2 -> 81.2 | Helped |
| Perform self-attention before cross-attention in the decoder (ablation) | LMIM · Union14M ablation | Avg accuracy: 80.2 -> 81.2 | Helped |
| Use MAE features as the reconstruction target instead of pixels or random features (ablation) | LMIM · Union14M ablation | Avg accuracy: 76.4 / 71.5 -> 81.2 | Helped |
| Use random masking at ratio 0.8 rather than other tested masking strategies or ratios (ablation) | LMIM · Union14M ablation | Avg accuracy: 79.1 / 80.4 / 81.0 / 80.1 -> 81.2 | Helped |
| Pre-train with LMIM rather than SSM-S | ViT-S scene text recognizer · Union14M benchmark | Avg accuracy: 84.3 -> 84.7 | Helped |
| Pre-train LMIM for 20 epochs rather than the default training | ViT-S scene text recognizer · Union14M benchmark | Avg accuracy: 84.7 -> 86.3 | Helped |
| Use LMIM rather than SeqCLR | ViT-S scene text recognizer · Chinese benchmark | Avg accuracy: 80.3 -> 82.2 | Helped |
| Use LMIM rather than SSM-S with corrected labels | ViT-S scene text recognizer · Six common benchmarks | Avg accuracy: 96.1 -> 97.0 | Helped |
Surprising findings
- Using a weak or medium-strength augmentation for the guidance view versus strong augmentation changes results only modestly, and too-strong augmentation hurts (Table 2).
Limitations
- Random fixed-patch masking is suboptimal given variable character counts in text images.
- Pre-training for masked image modeling is restricted to transformer architectures and not applicable to CNNs.
Experimental setup
Datasets: Union14M-U Union14M-L ARD UCTI-11M Chinese benchmark IIIT5K IC13 SVT IC15 SVTP CUTE80
Benchmarks: Union14M benchmark six common benchmarks (IIIT5K, IC13, SVT, IC15, SVTP, CUTE80) Chinese benchmark (Scene, Web, Document, Handwriting)
Metrics: word accuracy (WAICS) sentence-level accuracy average accuracy weighted average accuracy
Baselines: SeqCLR MIM DiG MAERec SSM MaskOCR CCD PARSeq MATRN ABINet
Builds on: MAE MAERec SeqCLR DiG ABINet
Closest papers by meaning
- OTE: Exploring Accurate Scene Text Recognition Using One Token (CVPR 2024 · similarity 0.89)
- Image as a Language: Revisiting Scene Text Recognition via Balanced, Unified and Synchronized Vision-Language Reasoning Network (AAAI 2024 · similarity 0.89)
- Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding (NeurIPS 2026 · similarity 0.88)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.88)
- Boosting Semi-Supervised Scene Text Recognition via Viewing and Summarizing (NeurIPS 2024 · similarity 0.87)
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.