Research map / Vision-language and multimodal / Multimodal LLMs

OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OCR-Reasoning, a 1,069-example human-annotated benchmark with step-by-step reasoning annotations for evaluating MLLMs on reasoning over text-rich images, finding that no model exceeds 50% accuracy.

Problem

Existing text-rich image benchmarks (DocVQA, ChartQA, OCRBench) mainly test text extraction rather than multi-step reasoning, leaving MLLM reasoning on text-rich images systematically unevaluated.

Approach

A benchmark of 1,069 single-image, human-annotated questions across 6 reasoning categories and 18 tasks, with both final answers and step-by-step reasoning annotations, scored by exact-match accuracy plus an LLM-as-judge reasoning-path score; the authors also show that OCR text input underperforms image input.

Key results

• DouBao-1.5-Vision-Pro achieves the highest final-answer accuracy at 46.8% overall, the only top closed model near 50%, and no evaluated model exceeds 50% accuracy.
• GLM-4.1V-Thinking-9B reaches 44.1% overall among open-source MLLMs, and OpenAI-o1 reaches 44.4%.
• Document-oriented MLLMs score at most 14.3% (TokenVL-8B), and Qwen2.5-VL-7B scores 15.7% on OCR-Reasoning while reaching 95.7 on DocVQA.

Surprising findings

• Replacing the image with PP-OCRv3 text output leads to lower accuracy than image input even for strong reasoning LLMs.
• Generic chain-of-thought prompting degrades VL-Rethinker-7B performance, attributed to conflict with its built-in reflection prompt.

Limitations

• Dataset of 1,069 examples is small, due to costly manual annotation.
• Reasoning-path scoring relies on LLM-as-judge, which has known biases; human validation only on four models.
• Restricted to single-image questions, excluding multi-image and multi-page settings.

Experimental setup

Datasets: InfoVQA DocVQA ChartQA CharXiv WildReceipt MME-Finance TextVQA

Benchmarks: OCR-Reasoning DocVQA ChartQA OCRBench TextVQA InfoVQA

Metrics: accuracy reasoning-path score

Baselines: OpenAI-o1 GPT-4o Gemini-2.0-Flash Claude-3.7-Sonnet DouBao-1.5-Vision-Pro Qwen2.5-VL InternVL3 Llama4-Scout-109B-A17B Kimi-VL-A3B-Thinking GLM-4.1V-Thinking-9B

Builds on: Chain-of-Thought prompting LLM-as-a-judge

Closest papers by meaning

• VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models (ICLR 2026 · similarity 0.89)
• GIR-Bench: Versatile Benchmark for Generating Images with Reasoning (ICLR 2026 · similarity 0.89)
• OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning (NeurIPS 2025 · similarity 0.89)
• ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.89)
• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.89)
• VLM-R³: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought (NeurIPS 2025 · similarity 0.89)