Research map / Vision-language and multimodal / Multimodal LLMs
OvisOCR: End-to-End Document Parsing via Aligning Specialized Perception with General Reasoning
Jun-Peng Jiang, Shiyin Lu, An-Yang Ji, Yinglun Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, Han-Jia Ye
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OvisOCR, a lightweight end-to-end MLLM that maps full-page document images directly to structured Markdown without crop-OCR-merge cascades or layout detection.
Problem
Document parsing of high-resolution pages typically relies on complex Crop-OCR-Merge cascades with layout detection; the paper targets strictly end-to-end parsing of full-page images to Markdown.
Approach
Directly maps full-page visual signals to structured Markdown with a lightweight MLLM, avoiding localized slicing and layout detection dependencies, and aligns specialized perception with general reasoning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add GRPO with a single global markdown edit-distance reward (ablation) | Ovis-SFT · OmniDocBench v1.5 | Overall: 92.30 -> 92.35 | No clear effect |
| replace global reward with decoupled per-category rewards (ablation) | Ovis-GRPO-MD · OmniDocBench v1.5 | Overall: 92.35 -> 92.84 | Improved |
| add decoupled rewards (vs SFT) (ablation) | Ovis-SFT · OmniDocBench v1.5 | FormulaCDM: 90.40 -> 90.70 | Improved |
| add decoupled rewards (vs global reward) (ablation) | Ovis-GRPO-MD · OmniDocBench v1.5 | TableTEDS: 89.69 -> 91.03 | Improved |
| add Semantic Auditor filtered data (ablation) | Ovis-GRPO-DR · OmniDocBench v1.5 | Overall: 92.84 -> 93.07 | Improved |
| add Semantic Auditor filtered data (ablation) | Ovis-GRPO-DR · OmniDocBench v1.5 | FormulaCDM: 90.70 -> 91.61 | Improved |
| add Semantic Auditor filtered data (ablation) | Ovis-GRPO-DR · OmniDocBench v1.5 | TableTEDS: 91.03 -> 90.80 | Hurt |
| add category-wise advantage normalization (GDPO) (ablation) | Ovis-GRPO-SA · OmniDocBench v1.5 | Overall: 93.07 -> 93.19 | Improved |
| add full RL alignment (GDPO) to SFT model (ablation) | Ovis-SFT · OmniDocBench v1.5 | Overall: 92.30 -> 93.19 | Improved |
Experimental setup
Benchmarks: OmniDocBench
Closest papers by meaning
- Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training (CVPR 2026 · similarity 0.90)
- DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.89)
- Boosting Document Parsing Efficiency and Performance with Coarse-to-Fine Visual Processing (CVPR 2026 · similarity 0.88)
- MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing (NeurIPS 2026 · similarity 0.87)
- Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation (NeurIPS 2026 · similarity 0.87)
- OmniParser: A Unified Framework for Text Spotting Key Information Extraction and Table Recognition (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
