Research map / Vision-language and multimodal / Multimodal LLMs
Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces KIE-HVQA, a benchmark for OCR hallucination under visual degradation, and a cold-start SFT plus GRPO framework with a degradation-aware OCR reward that teaches Qwen2.5-VL-7B to abstain on unrecognizable characters.
Problem
MLLMs hallucinate text content when document images are degraded (blur, occlusion, low contrast), defaulting to linguistic priors instead of abstaining when visual evidence is insufficient; no benchmark existed for measuring this OCR hallucination.
Approach
A three-tier character reward (clear characters must be output, partially obscured but recognizable ones output as anomalous, unrecognizable ones left blank) integrated into GRPO after a cold-start SFT on CoT data synthesized by chaining GPT-4o, an MLLM, and DeepSeek-R1, paired with a degradation-focused benchmark.
Key results
- Our 7B model with SFT+RL reaches 58.05% average score on KIE-HVQA, versus 30.21% for GPT-4o (1120).
- On not-clear characters, the model reaches 61.34% versus GPT-4o's 36.13%.
- On OCRBench-derived general categories (Scene/Doc/Info), the model scores 180/179/183, comparable to Qwen2.5-VL-7B (181/181/182).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reinforcement learning after SFT (ablation) | Qwen2.5-VL-7B-Instruct trained on KIE-HVQA · KIE-HVQA Average | Avg: 52.20 -> 58.05 | Helped |
| add the not-clear and final-character reward terms to the clear-character reward (ablation) | Qwen2.5-VL-7B-Instruct GRPO training · KIE-HVQA Average | Nc: 44.15 -> 61.34 | Helped |
| add the clear-character and not-clear reward terms to the final-character reward (ablation) | Qwen2.5-VL-7B-Instruct GRPO training · KIE-HVQA Average | Final: 54.24 -> 57.35 | Helped |
| use the composite reward rather than only the clear-character reward (ablation) | Qwen2.5-VL-7B-Instruct GRPO training · KIE-HVQA Average | Clr: 50.64 -> 55.45 | Helped |
| use the composite reward rather than only the final-character reward (ablation) | Qwen2.5-VL-7B-Instruct GRPO training · KIE-HVQA Average | Nc: 54.06 -> 61.34 | Helped |
| use the composite reward rather than only the clear-character reward (ablation) | Qwen2.5-VL-7B-Instruct GRPO training · KIE-HVQA Average | Final: 53.34 -> 57.35 | Helped |
Surprising findings
- Gemini2.5-pro outperforms GPT-4o and the Qwen2.5-VL-72B baseline on KIE-HVQA average score (35.03 vs. 30.21 and 21.53).
Limitations
- Degradations are simulated rather than naturally occurring.
- Reliance on GPT-4o and DeepSeek-R1 for cold-start CoT data and on GPT-4o and Qwen2.5-VL-72B for evaluation of degraded characters.
Experimental setup
Datasets: OCRBench WildReceipt TextOCR KIE-HVQA
Benchmarks: KIE-HVQA OCRBench
Metrics: character-level accuracy (clear, not clear, final) normalized Levenshtein similarity
Baselines: GPT-4o Claude3.5-Sonnet Claude3.7-Sonnet Gemini2.5-pro InternVL3-8B Qwen2.5-VL-8B Qwen2.5-VL-72B MiniCPM-o-2.6
Builds on: GRPO DeepSeek-R1 Qwen2.5-VL Easy-R1 LLaMA-Factory
Closest papers by meaning
- When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding (NeurIPS 2025 · similarity 0.90)
- Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy (ICML 2026 · similarity 0.90)
- Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning (NeurIPS 2025 · similarity 0.89)
- Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment (ICLR 2025 · similarity 0.89)
- MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation (ICLR 2025 · similarity 0.89)
- Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.