Research map / Vision-language and multimodal / Human and embodied modeling
SRFUND: A Multi-Granularity Hierarchical Structure Reconstruction Benchmark in Form Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SRFUND, a multilingual (8-language) form understanding benchmark with multi-granularity annotations (word-to-line, line-to-entity, entity classes, item tables, global hierarchical structure), and baselines from vision-only, text-only, and multimodal models.
Problem
Existing form understanding datasets (FUNSD, XFUND, etc.) provide only local, entity-level annotations and lack global hierarchical structure across words, text lines, entities, and item tables.
Approach
Refines FUNSD and XFUND with consistent word, text-line, entity, item-table, and global entity-link annotations, and defines five hierarchical tasks evaluated with an F1 metric that requires exact aggregation of element sets.
Key results
- LayoutXLM achieves the best average F1 on word-to-text-line merging (0.9260) and text-line-to-entity merging (0.9412) and entity classification (0.9248).
- GraphDoc achieves the best hierarchical structure recovery average F1 (0.7349), versus LayoutXLM (0.7013) and LiLT (0.6821).
- Item table localization is very hard: best average F1 is 0.3022 (LayoutXLM), with vision-only Cascade-RCNN at 0.1151.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a multi-modal document pre-trained model rather than a single-modal model | Word-to-text-line merging on SRFUND · Word-to-text-line merging (Avg.) | F1: 0.8194 -> 0.9260 | Helped |
| Use a multi-modal document pre-trained model rather than a vision-only model | Text-line-to-entity merging on SRFUND · Text-line-to-entity merging (Avg.) | F1: 0.7754 -> 0.9412 | Helped |
| Use a multi-modal document pre-trained model rather than a text-only model | Entity category classification on SRFUND · Entity category classification (Avg.) | F1: 0.9028 -> 0.9248 | Helped |
| Use a multi-modal document pre-trained model rather than a single-modal model | Item table localization on SRFUND · Item table localization (Avg.) | F1: 0.1514 -> 0.3022 | Helped |
| Align linguistic input granularity with entities by retaining sentence- or entity-level input | Document structure recovery models · Hierarchical structure recovery (Avg.) | F1: 0.7013 -> 0.7349 | Helped |
Surprising findings
- Item table localization remains very difficult for all modalities, with the best average F1 only 0.3022.
- Vision-only detectors perform poorly on text-line to entity merging compared to text-only models, showing weak semantic understanding of multi-line entities.
Limitations
- Item table localization accuracy is low across all tested models, indicating benchmark difficulty rather than solved capability.
- Only three model families per modality were evaluated, with no recent large multimodal LLMs.
Experimental setup
Datasets: FUNSD XFUND
Benchmarks: SRFUND
Metrics: F1-score
Baselines: YOLOX Cascade-RCNN DAB-DETR XLM-RoBerta InfoXLM LayoutXLM LiLT GraphDoc
Builds on: FUNSD XFUND CLEval Merger (GraphDCM)
Closest papers by meaning
- Graph-based Document Structure Analysis (ICLR 2025 · similarity 0.85)
- A Simple yet Effective Layout Token in Large Language Models for Document Understanding (CVPR 2025 · similarity 0.85)
- Relation-Rich Visual Document Generator for Visual Information Extraction (CVPR 2025 · similarity 0.84)
- SO-Bench: A Structural Output Evaluation of Multimodal LLM (CVPR 2026 · similarity 0.84)
- MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing (NeurIPS 2026 · similarity 0.84)
- COMM: Concentrated Margin Maximization for Robust Document-Level Relation Extraction (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.