Research map / Vision-language and multimodal / Human and embodied modeling
PreSTU: Pre-Training for Scene-Text Understanding
Jihyung Kil, Soravit Changpinyo, Xi Chen, Hexiang Hu, Sebastian Goodman, Wei-Lun Chao, Radu Soricut
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SplitOCR pre-training on CC15M | baseline ViT-B/16 + mT5-base · TextVQA test-std | accuracy: 44.8 -> 55.2 | Improved |
| add SplitOCR pre-training on CC15M | baseline · ST-VQA test | ANLS: 56.7 -> 65.5 | Improved |
| add SplitOCR pre-training | baseline · VQAv2 test-dev | accuracy: 74.8 -> 76.0 | Improved |
| add VQA second-stage pre-training after SplitOCR | PreSTU-SplitOCR · TextVQA test-std | accuracy: 55.2 -> 56.3 | Improved |
| add SplitOCR then CAP pre-training | PreSTU-SplitOCR · TextCaps test-std | CIDEr: 126.1 -> 139.1 | Improved |
| add SplitOCR pre-training | baseline · TextCaps test-std | CIDEr: 96.9 -> 126.1 | Improved |
| replace pure OCR pre-training with SplitOCR (stage 1) (ablation) | PreSTU · TextVQA val | accuracy: 49.9 -> 55.6 | Improved |
| add stage-1 SplitOCR pre-training, evaluated without OCR input at fine-tuning (ablation) | PreSTU · TextVQA val | accuracy: 13.7 -> 44.3 | Improved |
| add off-the-shelf OCR input at fine-tuning on top of SplitOCR+VQA+CAP (ablation) | PreSTU · TextVQA val | accuracy: 44.3 -> 56.7 | Improved |
| use 640 instead of 224 pre-training resolution (ablation) | PreSTU-SplitOCR · TextVQA val | accuracy: 47.1 -> 55.6 | Improved |
| increase pre-training data from 1% to 100% of CC15M (ablation) | PreSTU-SplitOCR · TextVQA val | accuracy: 42.3 -> 55.6 | Improved |
| use SplitOCR instead of CAP-only second-stage (ablation) | PreSTU · TextVQA val | accuracy: 49.3 -> 53.2 | Improved |
Closest papers by meaning
- TAP: Text-Aware Pre-Training for Text-VQA and Text-Caption (CVPR 2021 · similarity 0.92)
- Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting (ECCV 2022 · similarity 0.91)
- Vision-Language Pre-Training for Boosting Scene Text Detectors (CVPR 2022 · similarity 0.90)
- Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone (NeurIPS 2022 · similarity 0.89)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.88)
- InstructOCR: Instruction Boosting Scene Text Spotting (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
