Research map / Vision-language and multimodal / Multimodal LLMs
OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OCRBench v2, a bilingual benchmark of 23 text-centric tasks across 31 scenarios (10k human-verified QA pairs plus a 1,500-image private test set) and evaluates 18 LMMs, finding most score below 50.
Problem
Existing OCR benchmarks for large multimodal models are narrow in tasks and scenarios and are saturating, leaving text localization, handwritten content extraction, and logical reasoning underexplored.
Approach
A unified, large-scale bilingual benchmark spanning 8 core OCR capabilities and 23 tasks with six metric types, plus a private held-out test set to check generalization and contamination, and standardized [0,1000] coordinate prompting.
Key results
- InternVL3-14B achieves the highest public-set average of 52.6 on English tasks, while Gemini-Pro reaches 51.9 and most LMMs score below 50.
- Qwen2.5-VL-7B and InternVL3-14B score about 55-56 on the Chinese public tasks, with GPT-4o at 32.2 on Chinese public tasks averages.
- On the private English set, Gemini1.5-Pro leads with an average of 51.6 while GPT-4o scores 47.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate recognition on low-frequency rather than high-frequency text | InternVL3-14B · High-frequency vs low-frequency text | accuracy: 79.1% -> 46.7% | Hurt |
| Require exact answer-region localization in addition to answering | InternVL3-14B · VQA with position | IoU: 78.3% response accuracy; 12.9% IoU | Hurt |
| Rotate the DocVQA images | InternVL3-14B · DocVQA | performance: 90.9% -> 35.2%; 55.7% | Hurt |
| Require extracting the corresponding value for a given entity | InternVL3-14B · Unpaired entities matching vs key information extraction | accuracy: 94.4% -> 84.9% | Hurt |
| Evaluate complex reasoning question types | InternVL3-14B · Private test set reasoning tasks | accuracy: 72.9%, 83.0%, 69.2%, 56.5%, and 71.8% | Mixed |
Surprising findings
- InternVL3-14B reaches 78.3% accuracy on VQA with position but only 12.9% IoU for the answer region, indicating poor precise localization.
- Rotating DocVQA images reduces InternVL3-14B accuracy from 90.9% to 35.2%.
Limitations
- Private test set is not released, limiting external reproducibility.
- Focuses on single-image tasks and excludes multi-image or long multi-page document understanding.
- Benchmark may become contaminated as LMM pretraining data grows.
Experimental setup
Datasets: ICDAR2013 SVT IIIT5K ICDAR2015 SCUT-CTW1500 COCO-Text TotalText FUNSD SROIE DocVQA ChartQA TextVQA
Benchmarks: OCRBench v2 public OCRBench v2 private
Metrics: accuracy TEDS IoU F1 BLEU METEOR normalized edit distance ANLS L1-normalized counting score
Baselines: OCRBench GOT ABCNet TESTR CRNN ABINet ASTER MASTER SVTR
Builds on: OCRBench
Closest papers by meaning
- PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Tasks (CVPR 2026 · similarity 0.91)
- MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios (NeurIPS 2025 · similarity 0.90)
- OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning (ICLR 2026 · similarity 0.89)
- ASCIIEval: Benchmarking Models' Visual Perception in Text Strings via ASCII Art (ICLR 2026 · similarity 0.88)
- ConText: Driving In-context Learning for Text Removal and Segmentation (ICML 2025 · similarity 0.87)
- IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.