Research map / Vision-language and multimodal / Human and embodied modeling
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding
Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, Kristina Toutanova
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add screenshot parsing pretraining stage (vs. only extended warmup reading) (ablation) | Pix2Struct-Base pretraining · DocVQA validation / Widget Captioning / TextCaps validation | ANLS / CIDEr: DocVQA 67.8 vs 12.2; Widget Cap 137.5 vs 35.1; TextCaps 84.2 vs 24.2 | Improved |
| add warmup reading stage on BooksCorpus text before screenshot parsing (ablation) | Pix2Struct-Base pretraining · DocVQA validation / Widget Captioning / TextCaps validation | ANLS / CIDEr: DocVQA 67.8 vs 56.2; Widget Cap 137.5 vs 128.0; TextCaps 84.2 vs 71.7 | Improved |
| add masking of 50% of text during screenshot parsing (ablation) | Pix2Struct-Base pretraining · DocVQA validation / Widget Captioning / TextCaps validation | ANLS / CIDEr: DocVQA 67.8 vs 55.7; Widget Cap 137.5 vs 129.4; TextCaps 84.2 vs 77.4 | Improved |
| scale from Pix2Struct-Base to Pix2Struct-Large (1.3B) | Pix2Struct · all nine benchmarks (test) | task metric: e.g. ChartQA 56.0 to 58.6; DocVQA 72.1 to 76.6 (validation/test not specified) | Improved |
| use Pix2Struct (pixel-only, screenshot parsing) instead of Donut | pixel-only visual models · nine benchmarks (test) | task metric: Pix2Struct-Large beats Donut on all nine tasks, e.g. InfographicVQA 40.0 vs 11.6 | Improved |
| variable-resolution input (preserve aspect ratio) vs. fixed-resolution stretched or padded inputs (ablation) | Pix2Struct warmup stage · warmup full-sequence accuracy | full sequence accuracy | Improved |
| render text questions/choices as header on the image instead of a separate text channel | Pix2Struct finetuning for VQA · VQA tasks (OCR-VQA, ChartQA, DocVQA, InfographicVQA, AI2D) | task metric | No clear effect |
| pixel-only Pix2Struct-Large with no pipeline on TextCaps | TextCaps · TextCaps test | CIDEr: 95.5 vs pipeline SotA 160.4 and GIT2 145.0 | Hurt |
Closest papers by meaning
- Enhancing Vision-Language Pre-training with Rich Supervisions (CVPR 2024 · similarity 0.89)
- Harnessing Webpage UIs for Text-Rich Visual Understanding (ICLR 2025 · similarity 0.86)
- Read, Parse, Describe: Unified Document Parsing with Visual Element Description Generation (NeurIPS 2026 · similarity 0.86)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.86)
- Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding (CVPR 2026 · similarity 0.86)
- PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
