AI Paper Atlas

Research map / Vision-language and multimodal / Human and embodied modeling

Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding

Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova

ICML 2023 · Vision-language and multimodal › Human and embodied modeling · approach: Layout-aware document understanding · 47 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add screenshot parsing pretraining stage (vs. only extended warmup reading) (ablation)Pix2Struct-Base pretraining · DocVQA validation / Widget Captioning / TextCaps validationANLS / CIDEr: DocVQA 67.8 vs 12.2; Widget Cap 137.5 vs 35.1; TextCaps 84.2 vs 24.2Improved
add warmup reading stage on BooksCorpus text before screenshot parsing (ablation)Pix2Struct-Base pretraining · DocVQA validation / Widget Captioning / TextCaps validationANLS / CIDEr: DocVQA 67.8 vs 56.2; Widget Cap 137.5 vs 128.0; TextCaps 84.2 vs 71.7Improved
add masking of 50% of text during screenshot parsing (ablation)Pix2Struct-Base pretraining · DocVQA validation / Widget Captioning / TextCaps validationANLS / CIDEr: DocVQA 67.8 vs 55.7; Widget Cap 137.5 vs 129.4; TextCaps 84.2 vs 77.4Improved
scale from Pix2Struct-Base to Pix2Struct-Large (1.3B)Pix2Struct · all nine benchmarks (test)task metric: e.g. ChartQA 56.0 to 58.6; DocVQA 72.1 to 76.6 (validation/test not specified)Improved
use Pix2Struct (pixel-only, screenshot parsing) instead of Donutpixel-only visual models · nine benchmarks (test)task metric: Pix2Struct-Large beats Donut on all nine tasks, e.g. InfographicVQA 40.0 vs 11.6Improved
variable-resolution input (preserve aspect ratio) vs. fixed-resolution stretched or padded inputs (ablation)Pix2Struct warmup stage · warmup full-sequence accuracyfull sequence accuracyImproved
render text questions/choices as header on the image instead of a separate text channelPix2Struct finetuning for VQA · VQA tasks (OCR-VQA, ChartQA, DocVQA, InfographicVQA, AI2D)task metricNo clear effect
pixel-only Pix2Struct-Large with no pipeline on TextCapsTextCaps · TextCaps testCIDEr: 95.5 vs pipeline SotA 160.4 and GIT2 145.0Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy