Research map / Vision-language and multimodal / Human and embodied modeling
TextCaps: a Dataset for Image Captioning with Reading Comprehension
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, Amanpreet Singh
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add OCR token inputs (ablation) | M4C-Captioner · TextCaps val | CIDEr: 35.1 -> 89.6 | Improved |
| add pointer-network copying of OCR tokens (ablation) | M4C-Captioner · TextCaps val | CIDEr: 49.2 -> 89.6 | Improved |
| add spatial and visual OCR features to semantic-only features (ablation) | M4C-Captioner · TextCaps val | CIDEr: 69.0 -> 89.6 | Improved |
| add semantic and visual OCR features to spatial-only features (ablation) | M4C-Captioner · TextCaps val | CIDEr: 72.0 -> 89.6 | Improved |
| add semantic and spatial OCR features to visual-only features (ablation) | M4C-Captioner · TextCaps val | CIDEr: 84.0 -> 89.6 | Improved |
| add spatial OCR features to semantic and visual features (ablation) | M4C-Captioner · TextCaps val | CIDEr: 86.0 -> 89.6 | Improved |
| train on TextCaps instead of COCO | BUTD · TextCaps val | CIDEr: 24.2 -> 41.9 | Improved |
| train on TextCaps instead of COCO | AoANet · TextCaps val | CIDEr: 32.3 -> 42.7 | Improved |
| train on TextCaps instead of COCO | M4C-Captioner · TextCaps val | CIDEr: 30.3 -> 89.6 | Improved |
| use ground-truth OCR tokens for training and prediction | M4C-Captioner · TextCaps val | CIDEr: 89.6 -> 104.3 | Improved |
| train with GT OCRs but test with Rosetta OCRs | M4C-Captioner · TextCaps val | CIDEr: 89.6 -> 88.6 | No clear effect |
| train on COCO plus TextCaps instead of COCO alone | M4C-Captioner · COCO val (Karpathy split) | CIDEr: 112.2 -> 87.5 | Hurt |
Closest papers by meaning
- ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.88)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.88)
- Learning Interleaved Image-Text Comprehension in Vision-Language Large Models (ICLR 2025 · similarity 0.87)
- FlexCap: Describe Anything in Images in Controllable Detail (NeurIPS 2024 · similarity 0.87)
- CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness (NeurIPS 2025 · similarity 0.87)
- ViPCap: Retrieval Text-Based Visual Prompts for Lightweight Image Captioning (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
