Research map / Vision-language and multimodal / Human and embodied modeling
TAP: Text-Aware Pre-Training for Text-VQA and Text-Caption
Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, Jiebo Luo
Check your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply scene-text-aware pre-training with MLM, ITM, and RPP | M4C† · TextVQA val | accuracy: 44.50% to 49.91% | Improved |
| Add large-scale OCR-CC pre-training and other extra data | TAP · TextVQA val | accuracy: 49.91% to 54.71% | Improved |
| Apply TAP pre-training | M4C† · TextCaps val | CIDEr: 99.89 to 105.05 | Improved |
| Add scene-text words to MLM and ITM input (ablation) | MLM+ITM pre-training · TextVQA val | accuracy: 44.63 to 49.01 | Improved |
| Add MLM and ITM pre-training without scene-text words (ablation) | M4C · TextVQA val | accuracy: 44.50 to 44.63 | No clear effect |
| Add detected scene-text words without pre-training (ablation) | M4C · TextVQA val | accuracy: 44.48 to 44.50 | No clear effect |
| Add RPP with a single spatial relation class (ablation) | M4C · TextVQA val | accuracy: 44.50% to 46.42% | Improved |
| Add RPP to MLM and ITM (ablation) | MLM+ITM · TextVQA val | accuracy: 49.01% to 49.91% | Improved |
| Extend RPP from one spatial relation to 12 classes (ablation) | TAP · TextVQA val | accuracy: 49.91% to 50.17% | Improved |
| Add ST-VQA pre-training data (ablation) | TAP · TextVQA | accuracy: 49.91% to 50.57% | Improved |
| Add TextCaps pre-training data (ablation) | TAP with ST-VQA data · TextVQA | accuracy: 50.57% to 51.86% | Improved |
| Add OCR-CC as the pre-training data source (ablation) | TAP · TextVQA | accuracy: 52.10% | Improved |
Closest papers by meaning
- PreSTU: Pre-Training for Scene-Text Understanding (ICCV 2023 · similarity 0.92)
- Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.89)
- Simple is not Easy: A Simple Strong Baseline for TextVQA and TextCaps (AAAI 2021 · similarity 0.89)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.88)
- Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.87)
- InstructOCR: Instruction Boosting Scene Text Spotting (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
