Research map / Vision-language and multimodal / Human and embodied modeling
Relation-Rich Visual Document Generator for Visual Information Extraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RIDGE, a two-stage generator that uses an LLM to write hierarchical document content with entity categories and links, then a self-supervised LLM (LoRA-tuned LLaMA-3.1-8B) to place text into layouts from OCR-only data, and adds Hierarchical Structure Learning for MLLM fine-tuning on VIE.
Problem
Visual information extraction from relation-rich documents is hard due to layout diversity and scarce annotated data (especially open-set key-value and linking annotations); existing synthetic generators use manual templates or rules and layout generators ignore text content.
Approach
Generate document content in a Hierarchical Structure Text format with an LLM, then learn content-driven layout generation via self-supervised masked coordinate prediction on OCR-only data with randomly permuted entity order, and train MLLMs with an additional hierarchical structure parsing objective.
Key results
- Fine-tuning Qwen2-VL-7B with RIDGE raises FUNSD F1 from 59.89 to 66.48 in zero-shot VIE setting (+6.59).
- On EPHOIE, RIDGE-DS domain-specific generation lifts accuracy by 4.00 points and ANLS by 3.30 points over the RIDGE-initialized model.
- RIDGE pre-training improves LayoutLMv3 few-shot SER on XFUND-ZH, e.g., 0-shot 69.75 vs. no pretraining at 1-shot 52.02.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2-VL-7B with RIDGE | Qwen2-VL-7B · FUNSD | F1: 59.89 -> 66.48 | Helped |
| Fine-tune Qwen2-VL-7B with RIDGE | Qwen2-VL-7B · XFUND-ZH | F1: 62.08 -> 69.84 | Helped |
| Fine-tune Qwen2-VL-7B with RIDGE | Qwen2-VL-7B · CORD | F1: 82.71 -> 84.47 | Helped |
| Fine-tune Qwen2-VL-7B with RIDGE | Qwen2-VL-7B · CORD- | ANLS: 80.40 -> 85.53 | Helped |
| Fine-tune Qwen2-VL-7B with RIDGE | Qwen2-VL-7B · EPHOIE | ANLS: 86.52 -> 87.79 | Helped |
| Fine-tune Qwen2-VL-7B with RIDGE | Qwen2-VL-7B · POIE | ANLS: 93.51 -> 94.04 | Helped |
| Fine-tune Qwen2-VL-7B with RIDGE | Qwen2-VL-7B · SROIE- | ANLS: 97.50 -> 97.74 | Helped |
| Add domain-specific receipt documents | Qwen2-VL-7B+RIDGE · SROIE- | ANLS: 97.74 -> 98.05 | Helped |
| Add domain-specific exam cover page documents | Qwen2-VL-7B+RIDGE · EPHOIE | accuracy: 77.89 -> 80.91 | Helped |
| Pre-train LayoutLMv3 with RIDGE | LayoutLMv3 · FUNSD, 1-shot | F1: 45.42 -> 71.07 | Helped |
| Pre-train LayoutLMv3 with RIDGE | LayoutLMv3 · XFUND-ZH, 1-shot | F1: 52.02 -> 70.91 | Helped |
| Add Hierarchical Structure Learning to VIE training (ablation) | Qwen2-VL-7B · FUNSD | F1: 64.87 -> 66.48 | Helped |
Surprising findings
- RIDGE pre-training gives LayoutLMv3 a 62.77 score on FUNSD with zero fine-tuning shots, indicating zero-shot transfer of its semantic labels.
- Hierarchical Structure Learning gives additional gains of 1.61 on FUNSD and 2.52 on XFUND-ZH beyond VIE data alone.
Limitations
- Training focuses on form-like images; generalization to significantly distinct document types needs additional training.
- Evaluation of layouts uses downstream VIE performance because standard layout metrics like FID ignore text content.
Experimental setup
Datasets: RVL-CDIP FUNSD XFUND HUST-CELL DocVQA CORD POIE SROIE EPHOIE
Benchmarks: FUNSD XFUND-ZH CORD CORD- EPHOIE POIE SROIE-
Metrics: F1 ANLS accuracy
Baselines: LayoutLLM Monkey DocOwl-1.5 LLaVA-NeXT-Mistral-7B Qwen2-VL-7B DocILE SynthDoG DocSynth
Builds on: LayoutNUWA LayoutLLM Chain-of-Thought LayoutLMv3 Qwen2-VL
Closest papers by meaning
- DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding (CVPR 2025 · similarity 0.88)
- Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation (NeurIPS 2026 · similarity 0.86)
- SAIL: Sample-Centric In-Context Learning for Document Information Extraction (AAAI 2025 · similarity 0.86)
- An Autoregressive Text-to-Graph Framework for Joint Entity and Relation Extraction (AAAI 2024 · similarity 0.86)
- Graph-based Document Structure Analysis (ICLR 2025 · similarity 0.86)
- A Simple yet Effective Layout Token in Large Language Models for Document Understanding (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.