Research map / Vision-language and multimodal / Human and embodied modeling
InstructOCR: Instruction Boosting Scene Text Spotting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InstructOCR, an end-to-end scene text spotter that feeds attribute-based human-language instructions through a BERT text encoder alongside a ResNet-50 image encoder, and extends the pretrained model to scene text VQA.
Problem
Scene text spotting methods rely on image encoders and pre-trained text priors but do not exploit human language instructions to guide recognition; extending OCR pretraining to scene text VQA is also unexplored at small scale.
Approach
Instructions are automatically generated from existing word-level annotations using ten attribute-based templates (e.g., text length, capitalization, character presence), so no extra annotation is needed; a BERT text encoder encodes them and cross-attends with visual features in an SPTS-style point-sequence decoder.
Key results
- On Total-Text, InstructOCR reaches 77.1% lexicon-free Hmean (vs. SPTS-v2 75.5%) and 84.1% with full lexicon.
- On ICDAR2015 with strong lexicon, InstructOCR reaches 82.5% (87.5% with TextOCR and HierText added).
- On VQA, InstructOCR with instruction pretraining reaches 42.0% TextVQA and 45.8% ST-VQA using a 78M-parameter model, gaining 2.6 and 2.1 points from instructions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a text encoder to end-to-end training (ablation) | InstructOCR · ICDAR2015, Strong lexicon | Hmean: 86.5 -> 87.5 | Helped |
| train the text encoder specifically for end-to-end training (ablation) | InstructOCR · ICDAR2015, Strong lexicon | Hmean: 86.9 -> 87.1 | Helped |
| incorporate instructions into end-to-end training (ablation) | InstructOCR · ICDAR2015, Strong lexicon | Hmean: 87.1 -> 87.5 | Helped |
| incorporate instructions into end-to-end training (ablation) | InstructOCR · ICDAR2015, Weak lexicon | Hmean: 82.7 -> 84.2 | Helped |
| incorporate instructions into end-to-end training (ablation) | InstructOCR · ICDAR2015, Generic lexicon | Hmean: 79.1 -> 80.6 | No clear effect |
| incorporate instructions into end-to-end training (ablation) | InstructOCR · ICDAR2015, None | Hmean: 77.1 -> 78.9 | Helped |
| pretrain on 0.2M OCR data (ablation) | InstructOCR · ST-VQA | ANLS: 6.7 -> 43.7 | Helped |
| pretrain on 0.2M OCR data (ablation) | InstructOCR · TextVQA | accuracy: 5.6 -> 39.4 | Helped |
| incorporate instructions during pretraining (ablation) | InstructOCR · ST-VQA | ANLS: 43.7 -> 45.8 | Helped |
| incorporate instructions during pretraining (ablation) | InstructOCR · TextVQA | accuracy: 39.4 -> 42.0 | Helped |
| add TextOCR and HierText training data | InstructOCR · ICDAR2015, Strong lexicon | Hmean: 82.5 -> 87.5 | Helped |
Surprising findings
- Pretraining on only about 0.2M scene text images with a 78M ResNet-50+BERT model reaches VQA scores within a few points of much larger models in some cases, though it still trails InternVL, Monkey and Qwen-VL on TextVQA.
Limitations
- VQA experiments use only about 0.2M scene text images for pretraining and only TextVQA/ST-VQA for fine-tuning, so performance is far below large multimodal models.
- Comparisons with large VQA models are not apples-to-apples due to differing data and parameter counts.
- Instruction set is limited to ten attribute-based templates derived from existing annotations.
Experimental setup
Datasets: Total-Text ICDAR2015 ICDAR2013 Curved Synthetic Dataset 150k ICDAR2017 MLT TextOCR HierText TextVQA ST-VQA
Benchmarks: Total-Text ICDAR2015 ICDAR2013 TextVQA ST-VQA
Metrics: Hmean accuracy ANLS
Baselines: SPTS SPTS-v2 DeepSolo UNITS MANGO SRSTS TTS TAP GIT PreSTU WSCOB LLaVA1.5
Builds on: SPTS BERT ResNet
Closest papers by meaning
- ODM: A Text-Image Further Alignment Pre-training Approach for Scene Text Detection and Spotting (CVPR 2024 · similarity 0.87)
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.86)
- Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding (CVPR 2026 · similarity 0.86)
- MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios (NeurIPS 2025 · similarity 0.85)
- SemiETS: Integrating Spatial and Content Consistencies for Semi-Supervised End-to-end Text Spotting (CVPR 2025 · similarity 0.85)
- ViTEraser: Harnessing the Power of Vision Transformers for Scene Text Removal with SegMIM Pretraining (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.