AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

Document Understanding Dataset and Evaluation (DUDE)

Jordy Van Landeghem, Rubèn Tito, Łukasz Borchmann, Michał Pietruszka, Pawel Joziak, Rafal Powalski, Dawid Jurkiewicz, Mickael Coustaty, Bertrand Anckaert, Ernest Valveny, Matthew Blaschko, Sien Moens and 1 more

ICCV 2023 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 95 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
extend T5 max sequence length from 512 to 8192 tokens (ablation)T5-base fine-tuned on DUDE · DUDE testANLS (all): 37.41 -> 41.80Improved
extend T5-2D max sequence length from 512 to 8192 tokens (ablation)T5-2D base fine-tuned on DUDE · DUDE testANLS (all): 37.10 -> 42.10Improved
extend sequence length from 512 to 8192 tokens (ablation)T5-base fine-tuned on DUDE · DUDE testECE (all): 10.82 -> 17.33Hurt
add 2D layout embeddings to T5 (ablation)T5-base 8192 tokens · DUDE testANLS (all): 41.80 -> 42.10Improved
add 2D layout embeddings to T5 (ablation)T5-base 512 tokens · DUDE testANLS (all): 37.41 -> 37.10No clear effect
scale T5-2D from base (223M) to large (770M) (ablation)T5-2D, 8192 tokens · DUDE testANLS (all): 42.10 -> 46.06Improved
predict per page and take max-confidence answer instead of concatenating truncated tokens (ablation)BERT, BigBird, Longformer, LayoutLMv3, T5 MPDocVQA-init models · DUDE testECE: e.g. Longformer 22.21 -> 48.6Hurt
use per-page Max Conf. instead of Concat (ablation)BERT, LayoutLMv3, T5 MPDocVQA-init models · DUDE testANLS: BertQA 29.8 -> 32.18; BigBird 30.67 -> 29.38Mixed
add 4-shot prompt example to LLMs versus zero-shot (ablation)ChatGPT and GPT-3 davinci · DUDE diagnostic subsetANLS: ChatGPT 35.07 -> 41.89; GPT3 43.95 -> 47.04Improved
use large text-only LLM with long input instead of fine-tuned smaller modelGPT-3 / ChatGPT vs T5-2D large · DUDE diagnostic subset, extractive questionsANLS: T5-2D large 55.65; GPT3 4-shot 57.09No clear effect
use encoder-only models instead of generative modelsBigBird, BERT, Longformer, LayoutLMv3 · DUDE diagnostic subset, abstractive questionsANLS: below 10 ANLS on abstractiveHurt
use hierarchical HiVT5 with page tokens and vision featuresHiVT5 vs text-only T5 models · DUDE testANLS (all): 23.06 vs 37.41 (T5-base 512)Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy