Research map / Vision-language and multimodal / Multimodal LLMs
Teaching VLMs to Admit Uncertainty in OCR from Lossy Visual Inputs
Shuhao Guan, Moule Lin, Cheng Xu, Jinman Zhao, Derek Greene
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Targets hallucination in VLM-based OCR on degraded document images by teaching models to express uncertainty rather than guess (abstract only; details unavailable).
Problem
VLMs replacing traditional OCR hallucinate fluent but incorrect text on visually degraded (lossy) document images without signaling uncertainty, which current accuracy-focused post-training encourages.
Approach
Post-training that trains VLMs to admit uncertainty in OCR outputs instead of guessing when visual input is lossy; the abstract does not specify the mechanism.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GRPO with UNC reward after UNC-tag SFT (ablation) | Qwen2.5-VL-7B · Blur-OCR | transcription accuracy (char-level): 0.830 -> 0.889 | Improved |
| Add GRPO after UNC-tag SFT (ablation) | Qwen2.5-VL-7B · Blur-OCR | UNC tag F1 (word-level): 0.286 -> 0.363 | Improved |
| Train SFT with UNC tags instead of plain transcripts (ablation) | Qwen2.5-VL-7B · Blur-OCR | transcription accuracy (char-level): 0.868 -> 0.830 | Hurt |
| Add pseudo-labeled cold start before GRPO (ablation) | Qwen2.5-VL-7B · Blur-OCR | UNC tag F1 (word-level): 0.009 -> 0.574 | Improved |
| Use pseudo-labeled error-based cold start instead of random-tag cold start (ablation) | Qwen2.5-VL-7B · Blur-OCR | UNC tag F1 (word-level): 0.298 -> 0.363 | Improved |
| Use word-level reward with char-level cold start instead of char-level reward (ablation) | Qwen2.5-VL-7B · Blur-OCR | UNC tag F1 (word-level): 0.363 -> 0.574 | Improved |
| Use word-level cold start with word reward instead of char cold start (ablation) | Qwen2.5-VL-7B · Blur-OCR | UNC tag F1 (word-level): 0.574 -> 0.378 | Hurt |
| Lower F-beta beta from 1.0 to 0.5 (ablation) | Qwen2.5-VL-7B · Blur-OCR | UNC tag F1 (word-level): 0.574 -> 0.685 | Improved |
| Set lambda to 1.0 instead of 0.9 (ablation) | Qwen2.5-VL-7B · Blur-OCR | transcription accuracy (word-level): 0.839 -> 0.767 | Hurt |
| Add tagging term to reward (lambda 0.9 vs 0.0) (ablation) | Qwen2.5-VL-7B · Blur-OCR | UNC tag F1 (word-level): 0.250 -> 0.685 | Improved |
| Add length-mismatch damping eta to reward (ablation) | Qwen2.5-VL-7B · Blur-OCR | transcription accuracy (word-level): 0.782 -> 0.839 | Improved |
| Apply method to a different backbone (InternVL2.5-8B) | InternVL2.5-8B · Blur-OCR | UNC tag F1 (word-level): 0.622 vs best baseline 0.491 | Improved |
Surprising findings
- State-of-the-art VLM OCR systems still hallucinate on degraded inputs, per the abstract.
Closest papers by meaning
- Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models (NeurIPS 2025 · similarity 0.89)
- Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCR (CVPR 2026 · similarity 0.87)
- PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Tasks (CVPR 2026 · similarity 0.87)
- VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
- Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere (NeurIPS 2025 · similarity 0.87)
- Probabilistic Language-Image Pre-Training (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
