Research map / Vision-language and multimodal / Multimodal LLMs
ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ConTextual, a 506-instance human-written benchmark for context-sensitive text-rich visual reasoning, and evaluates 14 LMMs and augmented LLMs against human performance using human, GPT-4, and GPT-4V judging.
Problem
Existing text-rich VQA benchmarks (TextVQA, STVQA, ESTVQA, etc.) mainly test OCR reading and do not require joint reasoning over text and visual context in an image, so the abilities of large multimodal models on context-sensitive text-rich visual reasoning are not measured.
Approach
Human-written imperative and question instructions over text-rich images, filtered so that they cannot be answered from OCR alone plus LLM reasoning, spanning eight visual scenarios; models are scored with human acceptance ratings and GPT-4 / GPT-4V reference-guided judging.
Key results
- GPT-4V achieves 49.3% human acceptance versus 80.1% for humans on a 280-sample subset, the best LMM result.
- GPT-4 judging has the highest correlation with humans among automatic metrics (ROC-AUC 85.9, Spearman 0.71).
- Best open model ShareGPT-4V-7B reaches 21.8% human acceptance; augmented GPT-4 with layout OCR + caption reaches 17.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add layout-aware OCR and image captions to GPT-4 (augmented LLM) versus vanilla OCR (ablation) | GPT-4 augmented LLM · ConTextual full dataset | GPT-4 evaluation accuracy: 15.9 -> 22.2 | Helped |
| Use GPT-4V instead of text-only augmented LLM pipelines | LMM vs augmented LLM · ConTextual 280-sample human evaluation | acceptance rating: 49.3 vs 17.2 | Helped |
| Improved image encoder and data diversity (LLaVA-Next-34B over LLaVA-1.5-13B) | open LMMs · ConTextual full dataset | GPT-4 evaluation accuracy: 20.8 -> 36.8 | Helped |
| Few-shot in-context examples (2 to 8 shots) (ablation) | Gemini-Pro-Vision, Idefics-9B, augmented LLM · ConTextual full dataset | GPT-4 evaluation accuracy | No clear effect |
| Use GPT-4 as automatic evaluator instead of BLEURT, Rouge-L, or BERTScore (ablation) | automatic evaluation · ConTextual 280-sample human correlation | ROC-AUC vs human judgments: 85.9 vs 72.9 (BLEURT) | Helped |
| Generate synthetic instructions with GPT-4V in-context prompting with negative demonstrations (ablation) | synthetic data generation for LMM evaluation · Misc natural scenes synthetic vs ConTextual examples | GPT-4 evaluation accuracy: LLaVA-1.5-13B 39.8 vs 29.7; GPT-4V 68.6 vs 48.0 | Mixed |
Surprising findings
- GPT-4V outperforms humans on the Abstract category (100.0 vs 75.5 on GPT-4 judging rows, per the fine-grained table), while all models struggle most on time reading.
- Rouge-L and BERTScore rank open models above GPT-4V despite humans scoring highest, indicating lexical metrics are unreliable here.
Limitations
- Human evaluation covers only a 280-sample subset and uses MTurk annotators mainly from the US, limiting diversity.
- Automatic evaluation with GPT-4V may be biased by the image, and GPT-4 judging may favor GPT-generated responses.
- Few-shot prompting did not substantially change performance, and the synthetic data pipeline is only a preliminary study on 200 images.
Experimental setup
Datasets: LAION-5B Rico OpenWebText InfographicVQA STVQA ESTVQA
Benchmarks: ConTextual
Metrics: response acceptance rate BLEURT Rouge-L BERTScore ROC-AUC Spearman correlation
Baselines: GPT-4 w/ Layout-aware OCR + Caption Gemini-Pro-Vision LLaVA-Next-34B LLaVA-1.5-13B ShareGPT-4V-7B Qwen-VL-7B mPLUG-Owl-v2-7B Idefics-9B
Builds on: LLaVA LLaVAR LAION-5B PaddleOCR
Closest papers by meaning
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)
- ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.90)
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.90)
- ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.89)
- VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.