Research map / Vision-language and multimodal / Human and embodied modeling
OmniParser: A Unified Framework for Text Spotting Key Information Extraction and Table Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniParser, a single encoder-decoder model that handles text spotting, key information extraction, and table recognition via a two-stage structured-points-then-polygon/content generation with spatial- and prefix-window prompting pre-training.
Problem
Visually-situated text parsing (text spotting, KIE, table recognition) is usually solved with task-specific architectures, causing modal isolation and complex workflows; unified models lack precise localization or strong performance.
Approach
A task-agnostic encoder-decoder that generates structured center-point sequences (with task structural tags) first, then decodes 16-point polygons and text content per point in parallel via region and content decoders, pre-trained with spatial-window and prefix-window prompting.
Key results
- On Total-Text, OmniParser reaches 84.0 end-to-end (None) and 88.9 (Full), surpassing prior SOTA by +1.5% without lexicon; on CTW1500 it gains +3.2% over prior best.
- On CORD, OmniParser achieves 84.8 field-level F1, outperforming prior generation-based end-to-end methods; on SROIE it gets 93.6 tree-edit accuracy.
- On PubTabNet, OmniParser reports 90.45 S-TEDS and 88.83 TEDS, outperforming EDD and Donut (reproduced) end-to-end.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use OmniParser instead of the strongest listed baseline | text spotting · Total-Text | end-to-end recognition (None): 82.5 -> 84.0 | Helped |
| use OmniParser instead of the strongest listed baseline | text spotting · CTW1500 | end-to-end recognition (None): 66.4 -> 66.8 | Helped |
| use OmniParser instead of UNITS | text spotting · ICDAR 2015 | end-to-end recognition (Generic lexicon): 80.3 -> 79.9 | Hurt |
| use OmniParser instead of Donut | KIE · CORD | field-level F1: 84.1 -> 84.8 | Helped |
| use OmniParser instead of Donut | table recognition · PubTabNet (validation) | TEDS: 88.3 -> 88.83 | Helped |
| add spatial-window prompting (ablation) | OmniParser text spotting · Total-Text | end-to-end recognition (None): 82.4 -> 82.9 | Helped |
| add prefix-window prompting (ablation) | OmniParser text spotting · Total-Text | end-to-end recognition (None): 82.4 -> 83.5 | Helped |
| add both spatial-window and prefix-window prompting (ablation) | OmniParser text spotting · Total-Text | end-to-end recognition (None): 82.4 -> 84.0 | Helped |
| replace Swin-B with ResNet50 (ablation) | OmniParser text spotting · Total-Text | end-to-end recognition (None): 84.0 -> 82.1 | Hurt |
| share decoder weights (ablation) | OmniParser text spotting · Total-Text | end-to-end recognition (None): 84.0 -> 82.5 | Hurt |
| increase Structured Points Decoder length from 1,500 to 2,000 (ablation) | OmniParser table recognition · PubTabNet (validation) | S-TEDS: 90.45 -> 90.45 | No clear effect |
| increase Content Decoder length from 200 to 300 (ablation) | OmniParser table recognition · PubTabNet (validation) | TEDS: 88.83 -> 88.96 | Helped |
Surprising findings
- Sharing parameters across the three decoders degraded text spotting performance, suggesting discrepancies among center-point, polygon, and content decoding subtasks.
- Pre-training on scene text data only, without document pre-training, still yields competitive KIE results on CORD and SROIE.
Limitations
- Relies on precise word-level point locations during training, which may not always be available.
- Does not parse non-text elements such as figures or charts.
- On ICDAR 2015 it trails UNITS under the generic lexicon.
Experimental setup
Datasets: Curved SynthText ICDAR 2013 ICDAR 2015 MLT 2017 Total-Text TextOCR HierText COCO Text Open Images V5 CTW1500 CORD SROIE
Benchmarks: Total-Text CTW1500 ICDAR 2015 CORD SROIE PubTabNet FinTabNet
Metrics: end-to-end F1 (None/Full, S/W/G lexicons) detection precision/recall/F1 field-level F1 tree-edit-distance accuracy TEDS S-TEDS
Baselines: DeepSolo UNITS SPTS TESTR Donut SeRum EDD ABCNet v2
Builds on: Pix2seq Donut DeepSolo UNITS Swin Transformer
Closest papers by meaning
- OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations (CVPR 2025 · similarity 0.86)
- Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation (NeurIPS 2026 · similarity 0.86)
- Towards Real-World Document Parsing via Realistic Scene Synthesis and Document-Aware Training (CVPR 2026 · similarity 0.84)
- Boosting Document Parsing Efficiency and Performance with Coarse-to-Fine Visual Processing (CVPR 2026 · similarity 0.84)
- MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing (NeurIPS 2026 · similarity 0.83)
- AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generation (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.