Research map / Vision-language and multimodal / Human and embodied modeling
LDP: Generalizing to Multilingual Visual Information Extraction by Language Decoupled Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LDP, which pre-trains a SAM-based vision-layout model (LDM) on English document images whose language content is erased with AnyText, then fine-tunes it for multilingual visual information extraction.
Problem
Visual information extraction from semi-structured documents is dominated by English pre-training data, so models generalize poorly to non-English languages.
Approach
Vision and layout features are largely language-invariant; decoupling language content from document images with a text-editing diffusion model (AnyText) during pre-training, plus a SAM-based model with multi-token information merging and late language-knowledge insertion via Sentence-BERT, yields cross-lingual generalization from English-only data.
Key results
- Cross-lingual zero-shot (fine-tune FUNSD, test XFUND): LDM reaches 66.07 average F1 vs. 60.61 for LiLT.
- Language-specific fine-tuning: LDM achieves 88.21 average F1 on XFUND w/o EN, above ESP's 86.76.
- LDM reaches 95.60 F1 on SIBR, above ESP's 95.27 and LayoutXLM's 94.72.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LDM instead of LiLT | Cross-lingual zero-shot VIE · XFUND w/o EN, fine-tuned on FUNSD | F1: 57.24 -> 62.90 | Helped |
| Use language-independent pre-training instead of ESP pre-training | LDM and ESP · XFUND w/o EN, language-specific fine-tuning | F1: 86.76 -> 88.21 | Helped |
| Add MTIM (ablation) | LDM with pre-training · FUNSD and XFUND zero-shot | F1: 84.55 -> 86.24; 57.25 -> 57.65 | Helped |
| Add MTIM without pre-training (ablation) | LDM · FUNSD and XFUND zero-shot | F1: 80.03 -> 54.58; 56.38 -> 44.76 | Hurt |
| Reduce decouple resolution from 2048 to 1024 (ablation) | LDM with MTIM · XFUND zero-shot from FUNSD | F1: 59.82 -> 60.68 | Helped |
| Reduce decouple resolution from 1024 to 768 (ablation) | LDM with MTIM · XFUND zero-shot from FUNSD | F1: 60.68 -> 59.95 | Hurt |
| Reduce decouple resolution from 1024 to 768 (ablation) | LDM with MTIM · FUNSD | F1: 85.54 -> 83.10 | Hurt |
| Add LKI to the decoder only (ablation) | LDM · FUNSD and XFUND | F1: 86.95 -> 87.37; 61.36 -> 61.51 | Helped |
| Add LKI to the classifier only (ablation) | LDM · FUNSD and XFUND | F1: 86.95 -> 88.23; 61.36 -> 62.90 | Helped |
| Add LKI to both decoder and classifier instead of classifier only (ablation) | LDM · FUNSD | F1: 88.23 -> 88.07 | Hurt |
| Add LKI to both decoder and classifier instead of classifier only (ablation) | LDM · XFUND | F1: 62.90 -> 62.99 | Helped |
Surprising findings
- Removing language content from images (decoupling) improved cross-lingual XFUND performance while only slightly lowering English accuracy.
- Without pre-training, adding MTIM hurt performance sharply (FUNSD 80.03 to 54.58 in the ablation).
Limitations
- AnyText distorts small dense text in document images, so decoupling is imperfect and depends on the chosen decouple resolution.
- Pre-training and ablations use English-only data and XFUND-centered evaluation with small test sets (50 samples per language).
Experimental setup
Datasets: DocBank RVL-CDIP FUNSD XFUND SIBR CORD
Benchmarks: FUNSD XFUND SIBR CORD
Metrics: F1 precision recall
Baselines: XLM-RoBERTa InfoXLM LayoutXLM LiLT ESP TRIE LayoutLMv3 UDOP
Builds on: SAM AnyText ESP LiLT LayoutXLM
Closest papers by meaning
- Semantic and Expressive Variations in Image Captions Across Languages (CVPR 2025 · similarity 0.87)
- Multilingual Diversity Improves Vision-Language Representations (NeurIPS 2024 · similarity 0.87)
- Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective (AAAI 2025 · similarity 0.87)
- Exploring Visual Pretraining for Learning Language Intelligence (CVPR 2026 · similarity 0.87)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.86)
- On Scaling Up a Multilingual Vision and Language Model (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.