Research map / Label-efficient and robust learning / Knowledge distillation
Accurate Scene Text Recognition with Efficient Model Scaling and Cloze Self-Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Empirical study of vision-encoder vs. text-decoder scaling for scene text recognition, plus Cloze Self-Distillation (teacher cloze-refined pseudolabels with KD into a student) and a differential cross-attention permutation language decoder, reaching SOTA on 10 of 11 STR benchmarks with real data only.
Problem
Scene text recognition is limited by sub-optimal encoder/decoder scaling choices, noisy labels in real training data, and decoder architectural limits in focusing on relevant context.
Approach
Shows decoder scaling yields larger gains than encoder scaling for STR, and proposes Cloze Self-Distillation, where a teacher's cloze-refined (full-context) predictions and pseudolabels supervise a student with limited permutation context, plus a differential cross-attention decoder with SwiGLU.
Key results
- CSD-D (ViT-B + PLD-Diff, 110.2M params) reaches AVG11 92.73% on Real data, versus CLIP4STR-L at 91.69% using 446M params.
- With RBU training data, CSD-D achieves AVG11 93.30% and wAVG6 97.62%, outperforming CLIP4STR-L's 97.42% wAVG6.
- Scaling the decoder from PLD-T to PLD-L gives about +1.56% average AVG11 across encoders on RBU, versus +1.16% from encoder scaling to ViT-B with 80.3M extra parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| scale the decoder from PLD-Tiny to PLD-Large (ablation) | ViT-T STR model · AVG11 (Real) | word accuracy: 90.08 -> 91.67 | Helped |
| scale the decoder from PLD-Tiny to PLD-Large (ablation) | ViT-T STR model · AVG11 (RBU) | word accuracy: 90.15 -> 91.93 | Helped |
| scale the decoder from PLD-Tiny to PLD-Large (ablation) | ViT-S STR model · AVG11 (Real) | word accuracy: 91.04 -> 92.36 | Helped |
| scale the decoder from PLD-Tiny to PLD-Large (ablation) | ViT-S STR model · AVG11 (RBU) | word accuracy: 91.28 -> 92.77 | Helped |
| scale the decoder from PLD-Tiny to PLD-Large (ablation) | ViT-B STR model · AVG11 (Real) | word accuracy: 90.81 -> 92.52 | Helped |
| scale the decoder from PLD-Tiny to PLD-Large (ablation) | ViT-B STR model · AVG11 (RBU) | word accuracy: 91.55 -> 92.98 | Helped |
| replace ground-truth labels with teacher pseudolabels (ablation) | ViT-T + PLD-Base · AVG11 (Real) | word accuracy: 91.6 -> 91.8 | Helped |
| add cloze soft-prediction knowledge distillation (ablation) | ViT-T + PLD-Base with pseudolabels · AVG11 (Real) | word accuracy: 91.8 -> 91.9 | Helped |
| replace ground-truth labels with teacher pseudolabels (ablation) | ViT-S + PLD-Base · AVG11 (Real) | word accuracy: 92.2 -> 92.4 | Helped |
| add cloze soft-prediction knowledge distillation (ablation) | ViT-S + PLD-Base with pseudolabels · AVG11 (Real) | word accuracy: 92.4 -> 92.5 | Helped |
| replace ground-truth labels with teacher pseudolabels (ablation) | ViT-B + PLD-Base · AVG11 (Real) | word accuracy: 92.0 -> 92.3 | Helped |
| add cloze soft-prediction knowledge distillation (ablation) | ViT-B + PLD-Base with pseudolabels · AVG11 (Real) | word accuracy: 92.3 -> 92.5 | Helped |
Surprising findings
- Contrary to prior work (PARSeq scaling study), increasing decoder depth/size improves STR accuracy, and decoder scaling gives larger gains than encoder scaling.
- With small decoders on Real data, ViT-B performs worse than ViT-S, which the authors attribute to label-noise sensitivity.
Limitations
- Results with RBU use additional synthetic-augmented data (Union14M-L subset), so comparisons depend on training data scale.
- Decoder does not update context or vision tokens across blocks, which was a design choice for compute reasons.
- Cloze refinement requires training a teacher first, adding a two-stage training cost.
Experimental setup
Datasets: COCO-Text RCTW17 Uber-Text ArT LSVT MLT19 TextOCR ReCTR OpenVINO IIIT5K SVT ICDAR13
Benchmarks: IC13 IIIT5K SVT CUTE80 IC15 SVTP HOST WOST ArT COCO-Text Uber-Text
Metrics: word accuracy AVG11 wAVG6 parameters GFLOPs
Baselines: CLIP4STR-B CLIP4STR-L CLIP4STR-H PARSeq ABINet VITSTR-S TRBA CRNN
Builds on: PARSeq Differential Transformer CLIP4STR Permutation Language Modeling
Closest papers by meaning
- OTE: Exploring Accurate Scene Text Recognition Using One Token (CVPR 2024 · similarity 0.88)
- What if We Only Use Real Datasets for Scene Text Recognition? Toward Scene Text Recognition With Fewer Labels (CVPR 2021 · similarity 0.87)
- An Empirical Study of Scaling Law for Scene Text Recognition (CVPR 2024 · similarity 0.87)
- Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition (CVPR 2025 · similarity 0.87)
- Boosting Semi-Supervised Scene Text Recognition via Viewing and Summarizing (NeurIPS 2024 · similarity 0.85)
- Image as a Language: Revisiting Scene Text Recognition via Balanced, Unified and Synchronized Vision-Language Reasoning Network (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.