Research map / Vision-language and multimodal / Multimodal LLMs

Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCR

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Consensus Entropy, a training-free metric measuring multi-VLM output agreement to verify OCR outputs, and CE-OCR, an ensemble-and-route framework that improves OCR accuracy while routing a small fraction of inputs to a stronger VLM.

Problem

VLMs make sample-level OCR errors that are hard to detect, and existing evaluation relies on benchmark averages or VLM-as-Judge, which cannot reliably verify visual-text consistency without labels.

Approach

Measure the entropy of pairwise output similarities across several independent VLMs: correct predictions converge in output space while errors diverge. Low consensus entropy triggers a weighted ensemble; high entropy routes the sample to a stronger VLM for rephrasing.

Key results

• On a 1K human-annotated PDF page set, CE achieves F1 of 48.0-51.3 vs 36.1-40.0 for VLM-as-Judge, a 15.2-point (42.1%) average gain for quality verification.
• CE-OCR routing (Qwen2.5-VL-72B rephrase) reaches 922 on OCRBench vs 888 for best single model (+8.2%), routing 7.3% of inputs.
• CE-Ensemble on OCRBench-V2 English OCR improves from 65.6 (InternVL2.5-26B) to 67.2 and Math from 47.7 to 50.1 (Gemini Pro baseline row).

Surprising findings

• Ensembles of small open-source VLMs (<10B) such as Ovis2-1B-based combinations can outperform larger closed-source SOTA models on OCRBench.
• ROVER discrete voting fails catastrophically on open-ended VLM tasks (Doc-VQA -42.1%, Math-VQA -92.0%).

Limitations

• Gains depend on ensemble diversity; identical-model sampling gives smaller gains and results vary across sampling runs.
• GPT-4o verbose outputs artificially inflate CE values, requiring output cleaning for monotonic behavior.
• Threshold theta (around 0.5) is calibrated on development data and task-dependent metric choice (edit vs cosine) is required.

Experimental setup

Datasets: OCRBench OCRBench-V2 CCOCR

Benchmarks: OCRBench OCRBench-V2 CCOCR

Metrics: F1 accuracy cumulative score

Baselines: VLM-as-Judge VL-Uncertainty Single Model Self-Consistency ROVER CE-Ensemble

Builds on: self-consistency ROVER VL-Uncertainty VLM-as-Judge

Closest papers by meaning

• PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Tasks (CVPR 2026 · similarity 0.87)
• OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoning (NeurIPS 2025 · similarity 0.86)
• MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarios (NeurIPS 2025 · similarity 0.85)
• OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning (ICLR 2026 · similarity 0.84)
• UPOCR: Towards Unified Pixel-Level OCR Interface (ICML 2024 · similarity 0.84)
• Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models (NeurIPS 2025 · similarity 0.84)