Research map / Vision-language and multimodal / Multimodal LLMs
Document Understanding Dataset and Evaluation (DUDE)
Jordy Van Landeghem, Rubèn Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Mickael Coustaty, Bertrand Anckaert, Ernest Valveny, Matthew Blaschko, Sien Moens and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| extend T5 max sequence length from 512 to 8192 tokens (ablation) | T5-base fine-tuned on DUDE · DUDE test | ANLS (all): 37.41 -> 41.80 | Improved |
| extend T5-2D max sequence length from 512 to 8192 tokens (ablation) | T5-2D base fine-tuned on DUDE · DUDE test | ANLS (all): 37.10 -> 42.10 | Improved |
| extend sequence length from 512 to 8192 tokens (ablation) | T5-base fine-tuned on DUDE · DUDE test | ECE (all): 10.82 -> 17.33 | Hurt |
| add 2D layout embeddings to T5 (ablation) | T5-base 8192 tokens · DUDE test | ANLS (all): 41.80 -> 42.10 | Improved |
| add 2D layout embeddings to T5 (ablation) | T5-base 512 tokens · DUDE test | ANLS (all): 37.41 -> 37.10 | No clear effect |
| scale T5-2D from base (223M) to large (770M) (ablation) | T5-2D, 8192 tokens · DUDE test | ANLS (all): 42.10 -> 46.06 | Improved |
| predict per page and take max-confidence answer instead of concatenating truncated tokens (ablation) | BERT, BigBird, Longformer, LayoutLMv3, T5 MPDocVQA-init models · DUDE test | ECE: e.g. Longformer 22.21 -> 48.6 | Hurt |
| use per-page Max Conf. instead of Concat (ablation) | BERT, LayoutLMv3, T5 MPDocVQA-init models · DUDE test | ANLS: BertQA 29.8 -> 32.18; BigBird 30.67 -> 29.38 | Mixed |
| add 4-shot prompt example to LLMs versus zero-shot (ablation) | ChatGPT and GPT-3 davinci · DUDE diagnostic subset | ANLS: ChatGPT 35.07 -> 41.89; GPT3 43.95 -> 47.04 | Improved |
| use large text-only LLM with long input instead of fine-tuned smaller model | GPT-3 / ChatGPT vs T5-2D large · DUDE diagnostic subset, extractive questions | ANLS: T5-2D large 55.65; GPT3 4-shot 57.09 | No clear effect |
| use encoder-only models instead of generative models | BigBird, BERT, Longformer, LayoutLMv3 · DUDE diagnostic subset, abstractive questions | ANLS: below 10 ANLS on abstractive | Hurt |
| use hierarchical HiVT5 with page tokens and vision features | HiVT5 vs text-only T5 models · DUDE test | ANLS (all): 23.06 vs 37.41 (T5-base 512) | Hurt |
Closest papers by meaning
- MosaicDoc: A Large-Scale Bilingual Benchmark for Visually Rich Document Understanding (AAAI 2026 · similarity 0.87)
- MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations (NeurIPS 2024 · similarity 0.87)
- Docopilot: Improving Multimodal Models for Document-Level Understanding (CVPR 2025 · similarity 0.87)
- Graph-based Document Structure Analysis (ICLR 2025 · similarity 0.86)
- InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions (AAAI 2024 · similarity 0.85)
- DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
