Research map / Label-efficient and robust learning / Recognition and long-tail
Knowledge-Enhanced Historical Document Segmentation and Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KESAR, an abductive-learning method that injects first-order-logic structural knowledge into historical document segmentation (CRAFT) and recognition (ResNet) models, with abductive matching and joint optimization between them.
Problem
OCR of historical document images is hard due to distorted text lines, very large character sets with many uncommon characters, and scarce labeled data, so data-driven segmentation and recognition perform poorly.
Approach
Uses the Abductive Learning framework to revise pseudo-labels (character boxes and characters) with FOL background knowledge (square characters, similar aspect ratios within a line, no horizontal adjacency in vertical text), via a dynamic-programming abductive matching that aligns predictions to text-line labels, and an Over-Segmentation and Recombination algorithm for joint training of segmenter
Key results
- On MTH, KESAR reaches 93.2% text-line segmentation F-measure, 5.4 points above PSENet and 13.0 above FCENet.
- On GBACHD, KESAR achieves 94.2% segmentation F-measure and 0.875 1-N.E.D. recognition, at least 0.153 above RobustScanner and ABINet.
- Recognition on MTH reaches 0.924 1-N.E.D., 0.093 above its initial performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add structural knowledge to revise pseudo-character bounding boxes (ablation) | CRAFT segmentation model · MTH | F-measure: 87.8 -> 93.1 | Helped |
| add structural knowledge to revise pseudo-character bounding boxes (ablation) | CRAFT segmentation model · GBACHD | F-measure: 87.5 -> 93.5 | Helped |
| add joint optimization using the recognition model (ablation) | KESAR segmentation model · MTH | F-measure: 93.1 -> 93.2 | No clear effect |
| add joint optimization using the recognition model (ablation) | KESAR segmentation model · GBACHD | F-measure: 93.5 -> 94.2 | Helped |
| use KESAR instead of CRAFT | historical document segmentation · MTH | F-measure: 89.5 -> 93.2 | Helped |
| use KESAR instead of CRAFT | historical document segmentation · GBACHD | F-measure: 87.5 -> 94.2 | Helped |
| use KESAR instead of RobustScanner | historical document recognition · MTH | 1-N.E.D.: 0.905 -> 0.924 | Helped |
| use KESAR instead of RobustScanner | historical document recognition · GBACHD | 1-N.E.D.: 0.722 -> 0.875 | Helped |
Surprising findings
- On TKH, the comparison recognizers (RobustScanner, ABINet) slightly outperform KESAR (0.991/0.989 and 0.992/0.988 vs 0.970/0.988), attributed to KESAR training on predicted rather than ground-truth text lines.
Limitations
- Joint optimization gives little gain on MTH, where characters within a line are already well separated.
- Recognition on TKH trails strong baselines because training uses predicted rather than ground-truth text lines.
Experimental setup
Datasets: TKH MTH GBACHD
Benchmarks: TKH MTH GBACHD
Metrics: recall precision F-measure 1-N.E.D. successful abduction rate
Baselines: CRAFT PSENet FCENet RobustScanner ABINet
Builds on: Abductive Learning CRAFT ResNet
Closest papers by meaning
- Reference-Based Post-OCR Processing with LLM for Precise Diacritic Text in Historical Document Recognition (AAAI 2025 · similarity 0.83)
- Reading between the Lines: Image-Based Order Detection in OCR for Chinese Historical Documents (AAAI 2024 · similarity 0.82)
- ConText: Driving In-context Learning for Text Removal and Segmentation (ICML 2025 · similarity 0.81)
- DocSAM: Unified Document Image Segmentation via Query Decomposition and Heterogeneous Mixed Learning (CVPR 2025 · similarity 0.81)
- Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmark (CVPR 2025 · similarity 0.80)
- Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding (NeurIPS 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.