Research map / Vision-language and multimodal / CLIP and zero-shot
Enhancing Vision-Language Pre-training with Rich Supervisions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S4, a pre-training paradigm that renders CommonCrawl pages into screenshots and uses 10 HTML-derived supervised tasks to pre-train a Pix2Struct-style image-to-text model, improving document, UI, and detection downstream tasks.
Problem
Screenshot-based vision-language pre-training mostly uses HTML or raw text as targets, leaving fine-grained automatically obtainable supervision (layout, element grounding, tables) under-explored for transferring to downstream chart, UI, and detection tasks.
Approach
Render web pages with Playwright and extract rich annotations from the DOM tree (text, boxes, attributes, node relations, tables, layout) to build ten pre-training tasks that mirror downstream tasks, grouped into S4NL (language outputs) and S4Loc (localization outputs) schemes.
Key results
- With 15M S4 data, S4NL improves over Pix2Struct baseline by 2.9 on ChartQA, 6.8 on RefExp classification, 1.4 on Widget Captioning, 1.2 on Screen Summarization, and 1.0 on WebSRC.
- S4Loc raises RefExp candidate-free accuracy from 52.7 to 83.6 (2M data) and ICDAR 2019 AP50 from 3.3 to 70.7 (2M), reaching 79.4 with 15M data.
- Abstract reports up to 76.1% improvement on Table Detection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train with S4NL instead of Screen Parsing on 15M screenshots | Pix2Struct-base · ChartQA | Relaxed Match: 52.1 -> 55.0 | Helped |
| Pre-train with S4NL instead of Screen Parsing on 15M screenshots | Pix2Struct-base · RefExpcls | accuracy: 88.1 -> 94.9 | Helped |
| Pre-train with S4NL instead of Screen Parsing on 15M screenshots | Pix2Struct-base · Widget Captioning | Cider: 129.2 -> 130.6 | Helped |
| Pre-train with S4NL instead of Screen Parsing on 15M screenshots | Pix2Struct-base · Screen2words | Cider: 104.5 -> 105.7 | Helped |
| Pre-train with S4NL instead of Screen Parsing on 15M screenshots | Pix2Struct-base · WebSRC | Exact Match: 60.1 -> 61.1 | Helped |
| Pre-train with S4Loc instead of Screen Parsing | Pix2Struct-base · RefExp candidate free | accuracy: 52.7 -> 84.3 | Helped |
| Pre-train with S4Loc instead of Screen Parsing | Pix2Struct-base · ICDAR 2019 Modern Subset | AP 50: 3.3 -> 79.4 | Helped |
| Add Screenshot Titling to Screen Parsing pre-training (ablation) | Pix2Struct-base · Screen2words | Cider: 101.3 -> 101.4 | No clear effect |
| Add Attribute Prediction to Screenshot Titling and Screen Parsing (ablation) | Pix2Struct-base · Widget Captioning | Cider: 129.5 -> 128.3 | Hurt |
| Add Attribute Prediction to Screenshot Titling and Screen Parsing (ablation) | Pix2Struct-base · RefExpcls | accuracy: 87.8 -> 89.4 | Helped |
| Add Node Relation Prediction to the existing natural-language pre-training tasks (ablation) | Pix2Struct-base · RefExpcls | accuracy: 89.4 -> 92.7 | Helped |
| Add Table Parsing to the existing S4NL tasks (ablation) | Pix2Struct-base · Screen2words | Cider: 101.1 -> 103.2 | Helped |
Surprising findings
- Further pre-training Pix2Struct on 2M S4 data with only screen parsing slightly harms performance versus the original Pix2Struct checkpoint, attributed to data distribution shift and smaller batch size.
Limitations
- Baseline comparisons use Pix2Struct private-data checkpoint which cannot be exactly reproduced; S4 pre-training batch size smaller (128 vs 2048) due to compute.
- Image-encoder-text-decoder architecture requires high resolution input, increasing memory usage.
Experimental setup
Datasets: CommonCrawl ChartQA WebSRC Screen2words Widget Captioning RefExp PubLayNet PubTables-1M ICDAR 2019
Benchmarks: ChartQA WebSRC Screen2words Widget Captioning RefExp PubLayNet PubTables-1M ICDAR 2019
Metrics: relaxed accuracy exact match BLEU CIDEr classification accuracy AP50
Baselines: Pix2Struct Donut DETR DiT-B
Builds on: Pix2Struct MarkupLM Donut
Closest papers by meaning
- Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gain (ICML 2026 · similarity 0.88)
- TIPS: Text-Image Pretraining with Spatial awareness (ICLR 2025 · similarity 0.87)
- Tag2Text: Guiding Vision-Language Model via Image Tagging (ICLR 2024 · similarity 0.87)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.87)
- Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models (CVPR 2026 · similarity 0.87)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.